2023-04-25 06:53:36,789 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753 2023-04-25 06:53:36,803 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-25 06:53:36,820 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-25 06:53:36,821 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/cluster_91fce01a-9db1-5e0b-1ff6-3a8bbb37f430, deleteOnExit=true 2023-04-25 06:53:36,821 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-25 06:53:36,821 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/test.cache.data in system properties and HBase conf 2023-04-25 06:53:36,822 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/hadoop.tmp.dir in system properties and HBase conf 2023-04-25 06:53:36,822 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/hadoop.log.dir in system properties and HBase conf 2023-04-25 06:53:36,823 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-25 06:53:36,823 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-25 06:53:36,824 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-25 06:53:36,942 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-25 06:53:37,372 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-25 06:53:37,378 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-25 06:53:37,378 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-25 06:53:37,379 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-25 06:53:37,379 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-25 06:53:37,380 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-25 06:53:37,380 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-25 06:53:37,381 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-25 06:53:37,381 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-25 06:53:37,382 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-25 06:53:37,382 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/nfs.dump.dir in system properties and HBase conf 2023-04-25 06:53:37,383 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/java.io.tmpdir in system properties and HBase conf 2023-04-25 06:53:37,383 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-25 06:53:37,383 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-25 06:53:37,384 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-25 06:53:38,001 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-25 06:53:38,006 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-25 06:53:38,313 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-25 06:53:38,495 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-25 06:53:38,516 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-25 06:53:38,556 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-25 06:53:38,595 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/java.io.tmpdir/Jetty_localhost_34271_hdfs____gwcb3p/webapp 2023-04-25 06:53:38,739 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34271 2023-04-25 06:53:38,755 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-25 06:53:38,755 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-25 06:53:39,318 WARN [Listener at localhost/44945] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-25 06:53:39,397 WARN [Listener at localhost/44945] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-25 06:53:39,424 WARN [Listener at localhost/44945] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-25 06:53:39,431 INFO [Listener at localhost/44945] log.Slf4jLog(67): jetty-6.1.26 2023-04-25 06:53:39,440 INFO [Listener at localhost/44945] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/java.io.tmpdir/Jetty_localhost_45507_datanode____.9evvem/webapp 2023-04-25 06:53:39,537 INFO [Listener at localhost/44945] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45507 2023-04-25 06:53:39,904 WARN [Listener at localhost/41663] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-25 06:53:40,474 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9c12fcbc87afd3bc: Processing first storage report for DS-b3cd1a70-6cbb-4318-8e33-5f757386289e from datanode 4823a6af-3343-4f18-aae6-ee75d58162a4 2023-04-25 06:53:40,476 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9c12fcbc87afd3bc: from storage DS-b3cd1a70-6cbb-4318-8e33-5f757386289e node DatanodeRegistration(127.0.0.1:46541, datanodeUuid=4823a6af-3343-4f18-aae6-ee75d58162a4, infoPort=34061, infoSecurePort=0, ipcPort=41663, storageInfo=lv=-57;cid=testClusterID;nsid=170375639;c=1682405618087), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-25 06:53:40,476 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9c12fcbc87afd3bc: Processing first storage report for DS-11bd239b-a518-4f77-ba2d-28ebe11a89bb from datanode 4823a6af-3343-4f18-aae6-ee75d58162a4 2023-04-25 06:53:40,477 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9c12fcbc87afd3bc: from storage DS-11bd239b-a518-4f77-ba2d-28ebe11a89bb node DatanodeRegistration(127.0.0.1:46541, datanodeUuid=4823a6af-3343-4f18-aae6-ee75d58162a4, infoPort=34061, infoSecurePort=0, ipcPort=41663, storageInfo=lv=-57;cid=testClusterID;nsid=170375639;c=1682405618087), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-25 06:53:40,517 DEBUG [Listener at localhost/41663] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753 2023-04-25 06:53:40,588 INFO [Listener at localhost/41663] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/cluster_91fce01a-9db1-5e0b-1ff6-3a8bbb37f430/zookeeper_0, clientPort=60416, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/cluster_91fce01a-9db1-5e0b-1ff6-3a8bbb37f430/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/cluster_91fce01a-9db1-5e0b-1ff6-3a8bbb37f430/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-25 06:53:40,605 INFO [Listener at localhost/41663] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60416 2023-04-25 06:53:40,615 INFO [Listener at localhost/41663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 06:53:40,618 INFO [Listener at localhost/41663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 06:53:41,293 INFO [Listener at localhost/41663] util.FSUtils(471): Created version file at hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767 with version=8 2023-04-25 06:53:41,293 INFO [Listener at localhost/41663] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/hbase-staging 2023-04-25 06:53:41,672 INFO [Listener at localhost/41663] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-25 06:53:42,157 INFO [Listener at localhost/41663] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-25 06:53:42,189 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 06:53:42,190 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-25 06:53:42,190 INFO [Listener at localhost/41663] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-25 06:53:42,190 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 06:53:42,191 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-25 06:53:42,354 INFO [Listener at localhost/41663] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-25 06:53:42,430 DEBUG [Listener at localhost/41663] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-25 06:53:42,547 INFO [Listener at localhost/41663] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36703 2023-04-25 06:53:42,558 INFO [Listener at localhost/41663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 06:53:42,560 INFO [Listener at localhost/41663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 06:53:42,583 INFO [Listener at localhost/41663] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36703 connecting to ZooKeeper ensemble=127.0.0.1:60416 2023-04-25 06:53:42,629 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:367030x0, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-25 06:53:42,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:36703-0x101ba1865200000 connected 2023-04-25 06:53:42,663 DEBUG [Listener at localhost/41663] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 06:53:42,664 DEBUG [Listener at localhost/41663] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 06:53:42,668 DEBUG [Listener at localhost/41663] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-25 06:53:42,676 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36703 2023-04-25 06:53:42,676 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36703 2023-04-25 06:53:42,677 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36703 2023-04-25 06:53:42,680 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36703 2023-04-25 06:53:42,680 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36703 2023-04-25 06:53:42,687 INFO [Listener at localhost/41663] master.HMaster(444): hbase.rootdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767, hbase.cluster.distributed=false 2023-04-25 06:53:42,760 INFO [Listener at localhost/41663] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-25 06:53:42,760 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 06:53:42,761 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-25 06:53:42,761 INFO [Listener at localhost/41663] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-25 06:53:42,761 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 06:53:42,761 INFO [Listener at localhost/41663] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-25 06:53:42,766 INFO [Listener at localhost/41663] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-25 06:53:42,769 INFO [Listener at localhost/41663] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41021 2023-04-25 06:53:42,771 INFO [Listener at localhost/41663] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-25 06:53:42,777 DEBUG [Listener at localhost/41663] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-25 06:53:42,778 INFO [Listener at localhost/41663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 06:53:42,781 INFO [Listener at localhost/41663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 06:53:42,783 INFO [Listener at localhost/41663] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41021 connecting to ZooKeeper ensemble=127.0.0.1:60416 2023-04-25 06:53:42,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:410210x0, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-25 06:53:42,788 DEBUG [Listener at localhost/41663] zookeeper.ZKUtil(164): regionserver:410210x0, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 06:53:42,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41021-0x101ba1865200001 connected 2023-04-25 06:53:42,789 DEBUG [Listener at localhost/41663] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 06:53:42,790 DEBUG [Listener at localhost/41663] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-25 06:53:42,791 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41021 2023-04-25 06:53:42,792 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41021 2023-04-25 06:53:42,792 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41021 2023-04-25 06:53:42,793 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41021 2023-04-25 06:53:42,793 DEBUG [Listener at localhost/41663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41021 2023-04-25 06:53:42,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:42,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-25 06:53:42,810 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:42,837 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-25 06:53:42,837 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-25 06:53:42,837 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:42,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-25 06:53:42,840 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,36703,1682405621453 from backup master directory 2023-04-25 06:53:42,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-25 06:53:42,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:42,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-25 06:53:42,845 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-25 06:53:42,845 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:42,847 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-25 06:53:42,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-25 06:53:42,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/hbase.id with ID: b57fcd59-35c3-4599-b304-b85663a94880 2023-04-25 06:53:42,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 06:53:43,009 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:43,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2aeef597 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:43,101 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a88ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:43,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:53:43,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-25 06:53:43,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-25 06:53:43,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-25 06:53:43,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:53:43,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:53:43,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 06:53:43,193 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store-tmp 2023-04-25 06:53:43,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:43,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-25 06:53:43,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:53:43,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:53:43,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-25 06:53:43,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:53:43,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:53:43,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 06:53:43,225 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/WALs/jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:43,251 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36703%2C1682405621453, suffix=, logDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/WALs/jenkins-hbase4.apache.org,36703,1682405621453, archiveDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/oldWALs, maxLogs=10 2023-04-25 06:53:43,309 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46541,DS-b3cd1a70-6cbb-4318-8e33-5f757386289e,DISK] 2023-04-25 06:53:43,321 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:53:43,396 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/WALs/jenkins-hbase4.apache.org,36703,1682405621453/jenkins-hbase4.apache.org%2C36703%2C1682405621453.1682405623262 2023-04-25 06:53:43,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46541,DS-b3cd1a70-6cbb-4318-8e33-5f757386289e,DISK]] 2023-04-25 06:53:43,397 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:53:43,398 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:43,402 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 06:53:43,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 06:53:43,484 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-25 06:53:43,492 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-25 06:53:43,526 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-25 06:53:43,541 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:43,548 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-25 06:53:43,550 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-25 06:53:43,568 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 06:53:43,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:53:43,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59325192, jitterRate=-0.11598575115203857}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-25 06:53:43,577 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 06:53:43,579 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-25 06:53:43,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-25 06:53:43,607 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-25 06:53:43,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-25 06:53:43,613 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-25 06:53:43,655 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 41 msec 2023-04-25 06:53:43,655 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-25 06:53:43,683 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-25 06:53:43,691 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-25 06:53:43,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-25 06:53:43,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-25 06:53:43,733 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-25 06:53:43,739 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-25 06:53:43,744 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-25 06:53:43,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:43,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-25 06:53:43,750 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-25 06:53:43,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-25 06:53:43,772 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-25 06:53:43,772 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-25 06:53:43,772 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:43,773 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,36703,1682405621453, sessionid=0x101ba1865200000, setting cluster-up flag (Was=false) 2023-04-25 06:53:43,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:43,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-25 06:53:43,802 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:43,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:43,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-25 06:53:43,817 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:43,821 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.hbase-snapshot/.tmp 2023-04-25 06:53:43,899 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(952): ClusterId : b57fcd59-35c3-4599-b304-b85663a94880 2023-04-25 06:53:43,905 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-25 06:53:43,912 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-25 06:53:43,912 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-25 06:53:43,916 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-25 06:53:43,920 DEBUG [RS:0;jenkins-hbase4:41021] zookeeper.ReadOnlyZKClient(139): Connect 0x4ba5f95e to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:43,929 DEBUG [RS:0;jenkins-hbase4:41021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45620e5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:43,929 DEBUG [RS:0;jenkins-hbase4:41021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@415d922e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-25 06:53:43,946 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-25 06:53:43,956 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 06:53:43,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 06:53:43,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 06:53:43,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 06:53:43,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-25 06:53:43,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:43,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-25 06:53:43,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:43,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682405653965 2023-04-25 06:53:43,967 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-25 06:53:43,969 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-25 06:53:43,969 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41021 2023-04-25 06:53:43,970 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-25 06:53:43,974 INFO [RS:0;jenkins-hbase4:41021] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-25 06:53:43,975 INFO [RS:0;jenkins-hbase4:41021] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-25 06:53:43,975 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1023): About to register with Master. 2023-04-25 06:53:43,975 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-25 06:53:43,978 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36703,1682405621453 with isa=jenkins-hbase4.apache.org/172.31.14.131:41021, startcode=1682405622759 2023-04-25 06:53:43,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-25 06:53:43,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-25 06:53:43,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-25 06:53:43,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-25 06:53:43,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-25 06:53:43,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:43,989 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-25 06:53:43,990 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-25 06:53:43,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-25 06:53:43,994 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-25 06:53:43,994 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-25 06:53:43,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682405623996,5,FailOnTimeoutGroup] 2023-04-25 06:53:43,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682405623997,5,FailOnTimeoutGroup] 2023-04-25 06:53:43,998 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:43,998 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-25 06:53:44,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,001 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,005 DEBUG [RS:0;jenkins-hbase4:41021] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-25 06:53:44,009 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-25 06:53:44,010 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-25 06:53:44,010 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767 2023-04-25 06:53:44,024 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:44,028 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-25 06:53:44,032 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/info 2023-04-25 06:53:44,033 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-25 06:53:44,035 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:44,036 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-25 06:53:44,039 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/rep_barrier 2023-04-25 06:53:44,040 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-25 06:53:44,042 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:44,042 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-25 06:53:44,045 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/table 2023-04-25 06:53:44,045 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-25 06:53:44,046 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:44,048 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740 2023-04-25 06:53:44,049 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740 2023-04-25 06:53:44,055 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:53:44,058 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-25 06:53:44,063 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:53:44,065 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62041223, jitterRate=-0.07551373541355133}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:53:44,065 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-25 06:53:44,065 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-25 06:53:44,065 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33653, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-25 06:53:44,065 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-25 06:53:44,066 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-25 06:53:44,066 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-25 06:53:44,066 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-25 06:53:44,067 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-25 06:53:44,067 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-25 06:53:44,072 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-25 06:53:44,072 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-25 06:53:44,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:44,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-25 06:53:44,097 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-25 06:53:44,101 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-25 06:53:44,103 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767 2023-04-25 06:53:44,103 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44945 2023-04-25 06:53:44,103 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-25 06:53:44,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 06:53:44,109 DEBUG [RS:0;jenkins-hbase4:41021] zookeeper.ZKUtil(162): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:44,109 WARN [RS:0;jenkins-hbase4:41021] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-25 06:53:44,110 INFO [RS:0;jenkins-hbase4:41021] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 06:53:44,110 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/WALs/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:44,111 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41021,1682405622759] 2023-04-25 06:53:44,120 DEBUG [RS:0;jenkins-hbase4:41021] zookeeper.ZKUtil(162): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:44,132 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-25 06:53:44,142 INFO [RS:0;jenkins-hbase4:41021] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-25 06:53:44,165 INFO [RS:0;jenkins-hbase4:41021] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-25 06:53:44,170 INFO [RS:0;jenkins-hbase4:41021] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-25 06:53:44,170 INFO [RS:0;jenkins-hbase4:41021] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,171 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-25 06:53:44,179 INFO [RS:0;jenkins-hbase4:41021] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,180 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,180 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,180 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,180 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,180 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,180 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-25 06:53:44,181 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,181 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,181 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,181 DEBUG [RS:0;jenkins-hbase4:41021] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 06:53:44,184 INFO [RS:0;jenkins-hbase4:41021] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,184 INFO [RS:0;jenkins-hbase4:41021] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,184 INFO [RS:0;jenkins-hbase4:41021] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,203 INFO [RS:0;jenkins-hbase4:41021] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-25 06:53:44,206 INFO [RS:0;jenkins-hbase4:41021] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41021,1682405622759-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,224 INFO [RS:0;jenkins-hbase4:41021] regionserver.Replication(203): jenkins-hbase4.apache.org,41021,1682405622759 started 2023-04-25 06:53:44,224 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41021,1682405622759, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41021, sessionid=0x101ba1865200001 2023-04-25 06:53:44,224 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-25 06:53:44,225 DEBUG [RS:0;jenkins-hbase4:41021] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:44,225 DEBUG [RS:0;jenkins-hbase4:41021] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41021,1682405622759' 2023-04-25 06:53:44,225 DEBUG [RS:0;jenkins-hbase4:41021] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:44,226 DEBUG [RS:0;jenkins-hbase4:41021] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:44,226 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-25 06:53:44,226 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-25 06:53:44,227 DEBUG [RS:0;jenkins-hbase4:41021] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:44,227 DEBUG [RS:0;jenkins-hbase4:41021] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41021,1682405622759' 2023-04-25 06:53:44,227 DEBUG [RS:0;jenkins-hbase4:41021] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-25 06:53:44,227 DEBUG [RS:0;jenkins-hbase4:41021] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-25 06:53:44,228 DEBUG [RS:0;jenkins-hbase4:41021] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-25 06:53:44,228 INFO [RS:0;jenkins-hbase4:41021] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-25 06:53:44,228 INFO [RS:0;jenkins-hbase4:41021] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-25 06:53:44,253 DEBUG [jenkins-hbase4:36703] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-25 06:53:44,258 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41021,1682405622759, state=OPENING 2023-04-25 06:53:44,268 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-25 06:53:44,270 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:44,270 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-25 06:53:44,275 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:53:44,342 INFO [RS:0;jenkins-hbase4:41021] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41021%2C1682405622759, suffix=, logDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/WALs/jenkins-hbase4.apache.org,41021,1682405622759, archiveDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/oldWALs, maxLogs=32 2023-04-25 06:53:44,367 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46541,DS-b3cd1a70-6cbb-4318-8e33-5f757386289e,DISK] 2023-04-25 06:53:44,371 INFO [RS:0;jenkins-hbase4:41021] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/WALs/jenkins-hbase4.apache.org,41021,1682405622759/jenkins-hbase4.apache.org%2C41021%2C1682405622759.1682405624346 2023-04-25 06:53:44,372 DEBUG [RS:0;jenkins-hbase4:41021] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46541,DS-b3cd1a70-6cbb-4318-8e33-5f757386289e,DISK]] 2023-04-25 06:53:44,470 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:44,474 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-25 06:53:44,478 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43026, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-25 06:53:44,493 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-25 06:53:44,493 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 06:53:44,499 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41021%2C1682405622759.meta, suffix=.meta, logDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/WALs/jenkins-hbase4.apache.org,41021,1682405622759, archiveDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/oldWALs, maxLogs=32 2023-04-25 06:53:44,524 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46541,DS-b3cd1a70-6cbb-4318-8e33-5f757386289e,DISK] 2023-04-25 06:53:44,530 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/WALs/jenkins-hbase4.apache.org,41021,1682405622759/jenkins-hbase4.apache.org%2C41021%2C1682405622759.meta.1682405624501.meta 2023-04-25 06:53:44,531 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46541,DS-b3cd1a70-6cbb-4318-8e33-5f757386289e,DISK]] 2023-04-25 06:53:44,531 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:53:44,533 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-25 06:53:44,552 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-25 06:53:44,558 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-25 06:53:44,565 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-25 06:53:44,565 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:44,565 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-25 06:53:44,565 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-25 06:53:44,569 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-25 06:53:44,571 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/info 2023-04-25 06:53:44,571 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/info 2023-04-25 06:53:44,572 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-25 06:53:44,573 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:44,573 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-25 06:53:44,575 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/rep_barrier 2023-04-25 06:53:44,575 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/rep_barrier 2023-04-25 06:53:44,576 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-25 06:53:44,579 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:44,579 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-25 06:53:44,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/table 2023-04-25 06:53:44,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/table 2023-04-25 06:53:44,582 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-25 06:53:44,583 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:44,585 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740 2023-04-25 06:53:44,589 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740 2023-04-25 06:53:44,596 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:53:44,600 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-25 06:53:44,605 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69167793, jitterRate=0.03068043291568756}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:53:44,605 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-25 06:53:44,616 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682405624459 2023-04-25 06:53:44,634 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-25 06:53:44,634 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-25 06:53:44,635 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41021,1682405622759, state=OPEN 2023-04-25 06:53:44,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-25 06:53:44,638 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-25 06:53:44,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-25 06:53:44,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41021,1682405622759 in 363 msec 2023-04-25 06:53:44,647 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-25 06:53:44,647 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 562 msec 2023-04-25 06:53:44,652 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 784 msec 2023-04-25 06:53:44,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682405624652, completionTime=-1 2023-04-25 06:53:44,653 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-25 06:53:44,653 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-25 06:53:44,725 DEBUG [hconnection-0x2781da71-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:44,732 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:44,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-25 06:53:44,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682405684753 2023-04-25 06:53:44,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682405744753 2023-04-25 06:53:44,754 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 100 msec 2023-04-25 06:53:44,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36703,1682405621453-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36703,1682405621453-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36703,1682405621453-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,791 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:36703, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,791 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-25 06:53:44,799 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-25 06:53:44,808 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-25 06:53:44,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-25 06:53:44,819 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-25 06:53:44,822 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 06:53:44,825 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 06:53:44,848 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:44,850 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac empty. 2023-04-25 06:53:44,851 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:44,851 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-25 06:53:45,275 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-25 06:53:45,277 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c31ed1c679bd183deae013f944f500ac, NAME => 'hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp 2023-04-25 06:53:45,290 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:45,290 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c31ed1c679bd183deae013f944f500ac, disabling compactions & flushes 2023-04-25 06:53:45,290 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:53:45,290 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:53:45,290 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. after waiting 0 ms 2023-04-25 06:53:45,290 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:53:45,290 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:53:45,290 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c31ed1c679bd183deae013f944f500ac: 2023-04-25 06:53:45,296 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 06:53:45,314 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682405625299"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405625299"}]},"ts":"1682405625299"} 2023-04-25 06:53:45,346 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 06:53:45,348 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 06:53:45,352 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405625348"}]},"ts":"1682405625348"} 2023-04-25 06:53:45,357 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-25 06:53:45,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c31ed1c679bd183deae013f944f500ac, ASSIGN}] 2023-04-25 06:53:45,375 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c31ed1c679bd183deae013f944f500ac, ASSIGN 2023-04-25 06:53:45,377 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c31ed1c679bd183deae013f944f500ac, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=false 2023-04-25 06:53:45,528 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c31ed1c679bd183deae013f944f500ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:45,529 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682405625528"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405625528"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405625528"}]},"ts":"1682405625528"} 2023-04-25 06:53:45,535 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c31ed1c679bd183deae013f944f500ac, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:53:45,696 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:53:45,696 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c31ed1c679bd183deae013f944f500ac, NAME => 'hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:53:45,697 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:45,698 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:45,698 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:45,698 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:45,700 INFO [StoreOpener-c31ed1c679bd183deae013f944f500ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:45,702 DEBUG [StoreOpener-c31ed1c679bd183deae013f944f500ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/info 2023-04-25 06:53:45,702 DEBUG [StoreOpener-c31ed1c679bd183deae013f944f500ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/info 2023-04-25 06:53:45,703 INFO [StoreOpener-c31ed1c679bd183deae013f944f500ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c31ed1c679bd183deae013f944f500ac columnFamilyName info 2023-04-25 06:53:45,704 INFO [StoreOpener-c31ed1c679bd183deae013f944f500ac-1] regionserver.HStore(310): Store=c31ed1c679bd183deae013f944f500ac/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:45,706 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:45,707 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:45,711 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c31ed1c679bd183deae013f944f500ac 2023-04-25 06:53:45,715 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:53:45,716 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c31ed1c679bd183deae013f944f500ac; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63754482, jitterRate=-0.04998418688774109}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-25 06:53:45,716 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c31ed1c679bd183deae013f944f500ac: 2023-04-25 06:53:45,718 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac., pid=6, masterSystemTime=1682405625689 2023-04-25 06:53:45,723 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:53:45,723 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:53:45,724 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c31ed1c679bd183deae013f944f500ac, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:45,725 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682405625724"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405625724"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405625724"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405625724"}]},"ts":"1682405625724"} 2023-04-25 06:53:45,733 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-25 06:53:45,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c31ed1c679bd183deae013f944f500ac, server=jenkins-hbase4.apache.org,41021,1682405622759 in 194 msec 2023-04-25 06:53:45,738 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-25 06:53:45,739 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c31ed1c679bd183deae013f944f500ac, ASSIGN in 361 msec 2023-04-25 06:53:45,740 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 06:53:45,740 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405625740"}]},"ts":"1682405625740"} 2023-04-25 06:53:45,744 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-25 06:53:45,750 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 06:53:45,753 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 940 msec 2023-04-25 06:53:45,823 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-25 06:53:45,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-25 06:53:45,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:53:45,872 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-25 06:53:45,896 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-25 06:53:45,907 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 49 msec 2023-04-25 06:53:45,918 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-25 06:53:45,935 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-25 06:53:45,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 22 msec 2023-04-25 06:53:45,956 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-25 06:53:45,962 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-25 06:53:45,962 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.117sec 2023-04-25 06:53:45,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-25 06:53:45,966 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-25 06:53:45,966 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-25 06:53:45,968 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36703,1682405621453-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-25 06:53:45,969 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36703,1682405621453-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-25 06:53:45,982 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-25 06:53:46,004 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x2d205425 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:46,009 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c01f436, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:46,025 DEBUG [hconnection-0x1d3af235-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:46,041 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:46,052 INFO [Listener at localhost/41663] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:53:46,079 INFO [Listener at localhost/41663] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=169, AvailableMemoryMB=5030 2023-04-25 06:53:46,095 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 06:53:46,099 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 06:53:46,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:53:46,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:53:46,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 06:53:46,118 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 06:53:46,121 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 06:53:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-25 06:53:46,128 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,130 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8 empty. 2023-04-25 06:53:46,132 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,132 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:53:46,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 06:53:46,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 06:53:46,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 06:53:46,556 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:53:46,558 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 266734ad02022a1c1fca6ab96809adb8, NAME => 'TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp 2023-04-25 06:53:46,573 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:46,573 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 266734ad02022a1c1fca6ab96809adb8, disabling compactions & flushes 2023-04-25 06:53:46,573 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,573 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,573 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. after waiting 0 ms 2023-04-25 06:53:46,573 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,573 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,573 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:46,578 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 06:53:46,580 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405626580"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405626580"}]},"ts":"1682405626580"} 2023-04-25 06:53:46,584 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 06:53:46,586 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 06:53:46,587 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405626586"}]},"ts":"1682405626586"} 2023-04-25 06:53:46,589 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 06:53:46,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=266734ad02022a1c1fca6ab96809adb8, ASSIGN}] 2023-04-25 06:53:46,605 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=266734ad02022a1c1fca6ab96809adb8, ASSIGN 2023-04-25 06:53:46,609 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=266734ad02022a1c1fca6ab96809adb8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=false 2023-04-25 06:53:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 06:53:46,761 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=266734ad02022a1c1fca6ab96809adb8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:46,762 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405626761"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405626761"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405626761"}]},"ts":"1682405626761"} 2023-04-25 06:53:46,765 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:53:46,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 266734ad02022a1c1fca6ab96809adb8, NAME => 'TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,931 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,934 DEBUG [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A 2023-04-25 06:53:46,934 DEBUG [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A 2023-04-25 06:53:46,935 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:53:46,935 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 266734ad02022a1c1fca6ab96809adb8 columnFamilyName A 2023-04-25 06:53:46,936 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.HStore(310): Store=266734ad02022a1c1fca6ab96809adb8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:46,936 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,939 DEBUG [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B 2023-04-25 06:53:46,939 DEBUG [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B 2023-04-25 06:53:46,939 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:53:46,939 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 266734ad02022a1c1fca6ab96809adb8 columnFamilyName B 2023-04-25 06:53:46,940 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.HStore(310): Store=266734ad02022a1c1fca6ab96809adb8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:46,940 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,942 DEBUG [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C 2023-04-25 06:53:46,942 DEBUG [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C 2023-04-25 06:53:46,943 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:53:46,943 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 266734ad02022a1c1fca6ab96809adb8 columnFamilyName C 2023-04-25 06:53:46,944 INFO [StoreOpener-266734ad02022a1c1fca6ab96809adb8-1] regionserver.HStore(310): Store=266734ad02022a1c1fca6ab96809adb8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:53:46,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:53:46,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:46,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:53:46,957 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 266734ad02022a1c1fca6ab96809adb8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69461623, jitterRate=0.03505884110927582}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:53:46,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:46,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., pid=11, masterSystemTime=1682405626919 2023-04-25 06:53:46,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:46,963 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=266734ad02022a1c1fca6ab96809adb8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:46,963 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405626963"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405626963"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405626963"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405626963"}]},"ts":"1682405626963"} 2023-04-25 06:53:46,970 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-25 06:53:46,970 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 in 201 msec 2023-04-25 06:53:46,973 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-25 06:53:46,974 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=266734ad02022a1c1fca6ab96809adb8, ASSIGN in 375 msec 2023-04-25 06:53:46,975 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 06:53:46,975 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405626975"}]},"ts":"1682405626975"} 2023-04-25 06:53:46,977 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 06:53:46,980 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 06:53:46,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 867 msec 2023-04-25 06:53:47,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 06:53:47,252 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-25 06:53:47,258 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x36d9637e to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,262 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@348b992a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,266 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,270 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,273 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 06:53:47,276 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 06:53:47,285 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x6b41e306 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,290 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2732b8c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,290 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x2411a862 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,298 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72ea4a65, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,298 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x377933e4 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,303 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1645b6b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,303 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x3e8196e0 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,309 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b52aa42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,310 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x1ca4a062 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,315 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1710407d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,316 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x75f5a5e6 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,323 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ef1151, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,323 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x71f9f426 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,360 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29fba56a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,361 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x5ae6c42b to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,366 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ec2fae8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,367 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x726fb449 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:53:47,372 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e44f1b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:53:47,381 DEBUG [hconnection-0x74b74e95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,386 DEBUG [hconnection-0x6ff4c9b4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,386 DEBUG [hconnection-0x62a93ad6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,386 DEBUG [hconnection-0x63093886-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,390 DEBUG [hconnection-0x55d4bc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,391 DEBUG [hconnection-0x1bd47f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,392 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,393 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,393 DEBUG [hconnection-0x3bac190f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,406 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,406 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,407 DEBUG [hconnection-0x7d721dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,407 DEBUG [hconnection-0x605ba4da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:53:47,412 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,423 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43114, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,424 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,424 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43130, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:47,437 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:53:47,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:47,472 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:47,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:47,479 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:47,479 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:47,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:47,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:47,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:47,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:47,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:47,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:47,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:47,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:47,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:47,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:47,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:47,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:47,529 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:47,529 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:47,529 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:47,530 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:47,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:47,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:47,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:47,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:47,535 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:47,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:47,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:47,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ecf11fe[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:47,540 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:47,540 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,545 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:47,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:47,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:47,546 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,546 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:47,550 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:47,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:47,557 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:47,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:47,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:47,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:47,557 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:47,566 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,566 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,566 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:47,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:47,568 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,570 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:47,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:47,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:47,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,586 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:47,586 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:47,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:47,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:47,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:47,586 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,588 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:47,589 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:47,589 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:47,690 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:47,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:47,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405687690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405687694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:47,728 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:47,728 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:47,729 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:47,729 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:47,729 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,730 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:47,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:47,731 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:47,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:47,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:47,734 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:47,734 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:47,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:47,737 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:47,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405687697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:47,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405687699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405687731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,737 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:47,742 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:47,742 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:47,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:47,747 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:47,747 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,747 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:47,747 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:47,759 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,759 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:47,760 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,760 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:47,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:47,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717ad6e0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:47,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:47,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:47,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,777 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:47,781 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:47,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:47,796 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:47,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:47,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:47,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:47,796 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:47,801 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,801 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,801 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:47,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:47,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:47,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:47,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,835 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:47,840 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,840 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,840 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:47,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:47,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:47,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:47,841 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:47,841 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:47,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405687875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405687875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405687876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:47,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405687880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405687875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,941 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:47,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:47,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:47,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:47,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:47,964 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:47,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:47,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:47,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:47,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:47,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:47,976 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,977 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,977 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:47,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:47,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:47,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:47,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:47,980 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:47,980 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:47,980 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:47,981 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:47,985 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12dca29[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:47,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:47,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:47,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:47,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:48,004 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:48,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:48,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:48,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:48,004 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:48,008 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,008 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,008 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,010 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,013 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,023 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:48,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:48,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,026 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,026 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,026 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,026 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:48,026 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/b20f22b34ac949d882a8fb1b1d8c8c0a 2023-04-25 06:53:48,126 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:48,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405688119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405688129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405688130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405688128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405688129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:48,141 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:48,141 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:48,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:48,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,147 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:48,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:48,153 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,153 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:48,153 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:48,153 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:48,168 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:48,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f52eaa[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:48,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:48,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:48,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:48,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:48,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:48,179 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:48,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:48,188 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:48,188 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:48,188 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:48,188 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:48,188 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:48,191 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,191 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,192 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,218 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:48,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:48,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:48,218 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,226 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:48,232 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:48,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,332 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:48,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:48,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:48,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:48,344 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:48,344 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:48,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:48,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,348 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,348 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:48,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:48,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:48,364 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:48,364 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:48,365 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,365 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:48,365 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:48,365 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:48,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:48,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:48,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,369 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,370 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,370 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:48,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f39f2b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:48,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:48,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,375 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:48,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:48,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:48,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:48,383 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:48,383 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:48,384 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:48,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:48,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:48,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:48,384 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:48,387 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,401 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:48,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:48,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,402 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:48,402 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:48,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405688435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405688436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405688438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405688443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405688445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,502 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:48,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:48,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:48,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:48,512 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:48,512 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:48,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:48,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,520 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,520 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,520 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,520 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:48,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:48,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,536 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:48,537 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:48,537 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:48,538 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:48,538 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,538 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:48,539 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:48,539 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,539 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:48,539 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:48,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65999c43[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:48,547 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:48,547 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,549 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:48,550 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,550 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:48,553 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:48,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:48,559 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:48,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:48,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:48,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:48,559 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:48,561 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,561 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,563 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,566 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,573 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:48,577 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,577 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,577 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,577 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,577 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:48,578 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:48,578 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:48,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,678 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:48,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:48,692 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/52feb1bd68e7435e92bbf8e591fe4d09 2023-04-25 06:53:48,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:48,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:48,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:48,697 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:48,698 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,700 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:48,700 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:48,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,702 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:48,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:48,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:48,732 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:48,732 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,733 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:48,733 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:48,733 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:48,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:48,738 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,738 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0a6e5b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:48,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:48,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,749 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,750 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,751 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:48,751 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:48,751 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:48,751 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:48,754 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:48,754 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,754 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:48,755 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:48,769 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:48,769 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:48,769 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:48,770 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:48,770 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:48,773 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,773 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,774 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,786 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:48,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:48,787 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:48,788 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:48,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,888 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:48,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:48,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:48,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:48,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:48,897 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:48,897 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:48,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:48,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,903 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:48,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:48,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:48,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:48,907 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,907 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:48,908 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:48,908 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:48,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:48,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a781ce[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:48,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:48,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,921 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:48,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:48,931 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:48,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:48,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:48,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:48,932 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:48,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:48,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,937 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:48,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:48,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:48,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:48,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:48,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:48,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:48,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:48,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:48,949 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:48,949 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:48,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:48,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405688946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405688946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405688951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:48,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405688951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:48,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:48,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405688956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,049 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:49,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:49,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:49,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:49,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:49,072 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:49,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:49,079 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:49,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:49,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,082 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:49,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:49,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:49,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:49,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:49,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:49,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:49,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:49,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:49,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,110 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f358d6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:49,113 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:49,113 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,115 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:49,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:49,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:49,118 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:49,119 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:49,119 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:49,119 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:49,119 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:49,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,135 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:49,135 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:49,135 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,135 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,136 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,136 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:49,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:49,136 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:49,136 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:49,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:49,135 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:49,136 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,141 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:49,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:49,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/d32f56e4f20d46718bf670b834b71426 2023-04-25 06:53:49,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/b20f22b34ac949d882a8fb1b1d8c8c0a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b20f22b34ac949d882a8fb1b1d8c8c0a 2023-04-25 06:53:49,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b20f22b34ac949d882a8fb1b1d8c8c0a, entries=250, sequenceid=13, filesize=16.2 K 2023-04-25 06:53:49,196 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-25 06:53:49,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/52feb1bd68e7435e92bbf8e591fe4d09 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/52feb1bd68e7435e92bbf8e591fe4d09 2023-04-25 06:53:49,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/52feb1bd68e7435e92bbf8e591fe4d09, entries=150, sequenceid=13, filesize=11.6 K 2023-04-25 06:53:49,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/d32f56e4f20d46718bf670b834b71426 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d32f56e4f20d46718bf670b834b71426 2023-04-25 06:53:49,236 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:49,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d32f56e4f20d46718bf670b834b71426, entries=150, sequenceid=13, filesize=11.6 K 2023-04-25 06:53:49,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:49,247 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:49,247 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:49,248 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:49,248 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:49,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 266734ad02022a1c1fca6ab96809adb8 in 1760ms, sequenceid=13, compaction requested=false 2023-04-25 06:53:49,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,252 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 06:53:49,253 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:49,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:49,253 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:49,254 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,254 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:49,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:49,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:49,288 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:49,288 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:49,288 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:53:49,289 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:49,289 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:49,289 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:49,289 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:49,289 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:49,290 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:49,352 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/b05e733f88bc40b0838ee32f8c48c084 2023-04-25 06:53:49,413 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/a6e1bf8ccefb40dc99f778c31822c46d 2023-04-25 06:53:49,474 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/3277b562c7154da894cbaa8c6fb07716 2023-04-25 06:53:49,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/b05e733f88bc40b0838ee32f8c48c084 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b05e733f88bc40b0838ee32f8c48c084 2023-04-25 06:53:49,514 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b05e733f88bc40b0838ee32f8c48c084, entries=150, sequenceid=33, filesize=11.6 K 2023-04-25 06:53:49,530 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/a6e1bf8ccefb40dc99f778c31822c46d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6e1bf8ccefb40dc99f778c31822c46d 2023-04-25 06:53:49,549 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6e1bf8ccefb40dc99f778c31822c46d, entries=150, sequenceid=33, filesize=11.6 K 2023-04-25 06:53:49,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/3277b562c7154da894cbaa8c6fb07716 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3277b562c7154da894cbaa8c6fb07716 2023-04-25 06:53:49,573 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3277b562c7154da894cbaa8c6fb07716, entries=150, sequenceid=33, filesize=11.6 K 2023-04-25 06:53:49,574 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 266734ad02022a1c1fca6ab96809adb8 in 286ms, sequenceid=33, compaction requested=false 2023-04-25 06:53:49,575 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:49,575 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5021b82f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:49,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:49,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,626 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:49,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:49,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:49,626 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:49,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:49,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:49,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:49,678 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:49,678 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:49,678 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:49,678 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:49,678 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:49,680 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,680 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,680 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:49,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:49,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:49,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:49,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:49,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:49,691 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:49,691 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:49,792 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:49,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:49,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:49,808 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:49,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:49,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:49,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:49,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,815 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:49,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:49,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:49,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:49,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:49,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:49,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:49,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:49,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:49,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:49,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:49,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:49,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:49,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9807cc6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:49,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:49,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:49,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:49,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:49,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:49,865 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:49,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:49,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:49,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:49,865 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:49,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:49,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,868 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:49,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:49,875 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:49,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:49,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:49,878 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:49,878 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:49,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:49,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:49,978 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:49,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:49,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:49,989 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:49,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:49,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:49,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:49,991 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,991 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:49,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:49,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:49,996 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:49,996 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:53:49,996 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:49,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:50,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,033 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b235c9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:50,042 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:50,042 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,044 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:50,046 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,047 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:50,047 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:50,047 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:50,050 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:50,050 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,050 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:50,061 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:50,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:50,088 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:50,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:50,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:50,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:50,088 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:50,091 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,091 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,091 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,095 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,100 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:50,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:50,100 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:50,100 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:50,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405690083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405630036, max timestamp=1682405630082Num uniques 859; 2023-04-25 06:53:50,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405690083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405630036, max timestamp=1682405630082Num uniques 859; 2023-04-25 06:53:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:53:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:53:50,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405630036, max timestamp=1682405630082Num uniques 859; 2023-04-25 06:53:50,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:53:50,149 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 06:53:50,208 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:50,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405690179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:50,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405690183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405690199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405690217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:50,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:50,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405690218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,223 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:50,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:50,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:50,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,228 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,228 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:50,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:50,248 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:50,248 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,248 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:50,248 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:50,248 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:50,248 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:50,260 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71e4eaa1[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:50,263 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:50,263 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,265 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,265 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,265 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:50,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:50,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:50,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:50,270 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,280 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-25 06:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:50,285 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:50,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:50,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:50,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:50,285 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:50,287 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-25 06:53:50,288 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,292 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:50,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:50,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,297 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,297 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:50,296 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,297 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:50,297 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405690318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405690325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405690325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,400 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:50,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:50,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:50,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:50,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:50,408 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:50,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:50,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:50,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:50,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:50,416 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,416 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:50,416 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:50,416 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:50,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c59c472[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:50,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:50,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:50,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:50,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:50,429 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:50,429 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,429 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:50,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405690425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405690428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:50,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:50,432 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:50,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:50,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:50,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:50,432 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:50,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,435 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,435 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,436 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,440 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,446 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,446 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,446 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:50,446 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:50,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:50,453 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:50,453 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:50,446 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,454 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,454 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,454 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/c72bbd08816a4971b307675ae808b38d 2023-04-25 06:53:50,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405690526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405690535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405690536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,553 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:50,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:50,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:50,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:50,581 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:50,581 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,582 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:50,582 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:50,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,586 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,586 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:50,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:50,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:50,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,589 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:50,590 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,590 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:50,591 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:50,591 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,594 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b5eb73[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:50,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:50,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,603 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:50,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:50,605 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:50,614 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:50,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:50,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:50,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:50,614 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:50,616 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,616 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,616 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,635 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:50,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:50,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:50,637 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:50,637 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:50,737 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:50,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:50,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405690735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405690735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:50,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:50,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:50,751 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:50,751 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,752 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:50,752 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:50,754 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,754 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,758 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,758 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,758 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,758 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:50,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:50,760 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:50,760 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:50,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:50,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:50,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:50,774 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@252a9e11[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:50,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:50,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,781 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:50,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,782 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:50,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:50,784 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:50,784 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,784 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:50,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:50,790 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:50,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:50,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:50,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:50,790 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:50,793 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,793 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,793 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,800 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,800 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,800 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:50,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,800 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,805 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,805 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,805 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,805 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,805 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,805 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:50,805 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:50,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:50,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405690841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405690844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405690846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,905 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:50,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:50,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:50,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:50,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:50,911 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:50,911 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,912 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:50,912 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:50,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,915 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,915 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,915 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,915 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:50,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:50,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:50,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:50,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:50,927 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:50,927 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,929 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73cdc9f7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:50,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:50,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,935 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:50,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:50,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:50,937 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:50,942 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:50,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:50,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:50,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:50,943 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:50,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:50,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:50,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,954 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:50,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:50,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:50,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:50,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:50,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:50,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:50,959 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:50,959 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,959 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:50,959 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:50,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:50,959 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:50,959 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:50,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/c9743864e82b4993aca4a7be4b778eec 2023-04-25 06:53:51,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/13125f1e8cff43d5bb5555df00b7db8c 2023-04-25 06:53:51,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/c72bbd08816a4971b307675ae808b38d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c72bbd08816a4971b307675ae808b38d 2023-04-25 06:53:51,059 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:51,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:51,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:51,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:51,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c72bbd08816a4971b307675ae808b38d, entries=200, sequenceid=44, filesize=13.9 K 2023-04-25 06:53:51,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:51,096 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:51,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:51,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:51,098 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/c9743864e82b4993aca4a7be4b778eec as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/c9743864e82b4993aca4a7be4b778eec 2023-04-25 06:53:51,106 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,106 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:51,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:51,124 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,124 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:51,124 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:51,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/c9743864e82b4993aca4a7be4b778eec, entries=150, sequenceid=44, filesize=11.6 K 2023-04-25 06:53:51,125 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,132 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/13125f1e8cff43d5bb5555df00b7db8c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/13125f1e8cff43d5bb5555df00b7db8c 2023-04-25 06:53:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1155e9a8[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:51,140 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:51,140 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,142 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:51,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:51,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/13125f1e8cff43d5bb5555df00b7db8c, entries=150, sequenceid=44, filesize=11.6 K 2023-04-25 06:53:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:51,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 266734ad02022a1c1fca6ab96809adb8 in 1153ms, sequenceid=44, compaction requested=true 2023-04-25 06:53:51,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:51,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:51,154 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:51,154 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:51,155 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:51,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:51,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:51,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:51,155 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:51,160 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:51,163 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:53:51,163 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,163 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/52feb1bd68e7435e92bbf8e591fe4d09, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6e1bf8ccefb40dc99f778c31822c46d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/c9743864e82b4993aca4a7be4b778eec] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=34.8 K 2023-04-25 06:53:51,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,167 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 52feb1bd68e7435e92bbf8e591fe4d09, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682405627475 2023-04-25 06:53:51,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,167 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:51,168 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,168 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:53:51,168 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,168 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,168 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a6e1bf8ccefb40dc99f778c31822c46d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682405627687 2023-04-25 06:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,168 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b20f22b34ac949d882a8fb1b1d8c8c0a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b05e733f88bc40b0838ee32f8c48c084, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c72bbd08816a4971b307675ae808b38d] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=41.7 K 2023-04-25 06:53:51,169 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c9743864e82b4993aca4a7be4b778eec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682405629968 2023-04-25 06:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,169 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b20f22b34ac949d882a8fb1b1d8c8c0a, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682405627464 2023-04-25 06:53:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,172 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b05e733f88bc40b0838ee32f8c48c084, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682405627687 2023-04-25 06:53:51,172 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,173 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c72bbd08816a4971b307675ae808b38d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682405629968 2023-04-25 06:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:51,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:51,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,178 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:51,178 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:51,178 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,178 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,178 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,208 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#9 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:51,227 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#10 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:51,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7687, totalHeapSize=17712, min timestamp=1682405630176, max timestamp=1682405631257Num uniques 1018; 2023-04-25 06:53:51,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46617, new segment data size=7687, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 06:53:51,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682405630176, max timestamp=1682405631246Num uniques 1018; 2023-04-25 06:53:51,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 06:53:51,278 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:51,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:51,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=71.56 KB heapSize=198.72 KB 2023-04-25 06:53:51,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:51,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:53:51,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:51,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:53:51,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:51,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 06:53:51,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:51,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:51,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:51,298 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:51,298 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:51,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:51,312 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,312 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,314 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,315 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,315 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:51,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:51,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:51,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:51,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,326 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:51,327 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:51,339 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:51,339 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:51,339 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,339 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:51,339 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:51,339 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,339 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:51,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:51,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,343 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:51,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:51,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18b26682[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:51,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:51,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:51,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:51,351 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:51,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:51,359 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:51,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:51,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:51,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:51,359 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:51,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,364 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405691358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405691360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405691363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:51,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:51,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,374 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,374 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:51,374 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,374 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:51,374 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405691365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,374 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405691365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405691467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405691467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,474 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:51,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:51,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:51,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405691473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405691477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405691476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:51,492 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:51,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:51,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:51,500 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,500 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:51,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:51,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:51,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:51,523 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6919b993[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:51,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:51,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,531 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,531 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:51,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:51,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:51,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:51,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:51,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:51,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:51,541 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:51,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:51,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:51,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:51,541 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:51,548 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,549 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,549 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,553 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,556 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:51,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:51,557 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:51,557 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:51,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:51,657 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:51,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:51,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:51,667 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 06:53:51,668 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-25 06:53:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:51,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:51,672 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:51,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:51,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:51,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,674 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:51,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405691675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405691676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:51,684 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,684 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:51,684 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:51,684 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:51,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74865736[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:51,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:51,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405691688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405691692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405691693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,700 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:51,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:51,703 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:51,707 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:51,707 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:51,707 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:51,707 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:51,707 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:51,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,717 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,730 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/df9a9ba8f1db411a934437e82e1032e1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/df9a9ba8f1db411a934437e82e1032e1 2023-04-25 06:53:51,731 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/0be88638532a47d4ad4d2c4b1b72a652 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0be88638532a47d4ad4d2c4b1b72a652 2023-04-25 06:53:51,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,731 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:51,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:51,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,732 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:51,732 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:51,732 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,732 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,732 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,732 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,732 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:51,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/dacd8fee0aca4d2db8744b1bed8cf932 2023-04-25 06:53:51,756 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into df9a9ba8f1db411a934437e82e1032e1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:51,756 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:51,756 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=13, startTime=1682405631152; duration=0sec 2023-04-25 06:53:51,757 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:51,757 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:51,757 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into 0be88638532a47d4ad4d2c4b1b72a652(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:51,757 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:51,757 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=13, startTime=1682405631154; duration=0sec 2023-04-25 06:53:51,757 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:51,759 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:51,759 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:53:51,759 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,759 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d32f56e4f20d46718bf670b834b71426, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3277b562c7154da894cbaa8c6fb07716, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/13125f1e8cff43d5bb5555df00b7db8c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=34.8 K 2023-04-25 06:53:51,760 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d32f56e4f20d46718bf670b834b71426, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682405627475 2023-04-25 06:53:51,760 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 3277b562c7154da894cbaa8c6fb07716, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682405627687 2023-04-25 06:53:51,761 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 13125f1e8cff43d5bb5555df00b7db8c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682405629968 2023-04-25 06:53:51,801 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#12 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:51,832 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:51,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:51,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:51,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:51,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:51,863 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:51,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:51,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:51,864 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,864 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,865 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:51,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:51,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:51,900 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,900 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:51,901 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:51,901 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:51,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,916 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/2aa4ac39663f47c080d0a0b3eb7a4d72 2023-04-25 06:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:51,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa0f8a7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:51,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:51,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:51,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:51,929 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,935 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/31f1c032f6bf403995f5bedb7c7b6568 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/31f1c032f6bf403995f5bedb7c7b6568 2023-04-25 06:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:51,938 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:51,938 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:51,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:51,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:51,939 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:51,941 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,941 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,941 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:51,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:51,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:51,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:51,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:51,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:51,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:51,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:51,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:51,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:51,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:51,956 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:51,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:51,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:51,957 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:51,958 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:51,961 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into 31f1c032f6bf403995f5bedb7c7b6568(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:51,961 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:51,961 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=13, startTime=1682405631154; duration=0sec 2023-04-25 06:53:51,961 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:51,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/747633842a354cdd91f152e0d2794aab 2023-04-25 06:53:51,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:51,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405691990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405691993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/dacd8fee0aca4d2db8744b1bed8cf932 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/dacd8fee0aca4d2db8744b1bed8cf932 2023-04-25 06:53:52,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/dacd8fee0aca4d2db8744b1bed8cf932, entries=150, sequenceid=83, filesize=11.6 K 2023-04-25 06:53:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405692000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405692001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405691990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/2aa4ac39663f47c080d0a0b3eb7a4d72 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/2aa4ac39663f47c080d0a0b3eb7a4d72 2023-04-25 06:53:52,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/2aa4ac39663f47c080d0a0b3eb7a4d72, entries=150, sequenceid=83, filesize=11.6 K 2023-04-25 06:53:52,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/747633842a354cdd91f152e0d2794aab as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/747633842a354cdd91f152e0d2794aab 2023-04-25 06:53:52,058 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:52,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:52,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:52,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/747633842a354cdd91f152e0d2794aab, entries=150, sequenceid=83, filesize=11.6 K 2023-04-25 06:53:52,083 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~198.67 KB/203440, currentSize=107.34 KB/109920 for 266734ad02022a1c1fca6ab96809adb8 in 798ms, sequenceid=83, compaction requested=false 2023-04-25 06:53:52,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:52,083 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:52,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:52,083 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:52,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,086 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:52,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:52,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:52,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:52,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:52,091 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:52,091 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:52,091 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:53:52,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:52,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:52,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:52,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:52,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:52,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:52,148 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/c3b574468fee4bd49f720b53ab0a0f94 2023-04-25 06:53:52,195 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5b0d5709c8a2455f93961781c91c6421 2023-04-25 06:53:52,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:52,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:52,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405692531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405692532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405692533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405692537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405692552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,629 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/12d68387f9714cdc9ee518abbf1b70b4 2023-04-25 06:53:52,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405692642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405692646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405692646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405692651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,656 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/c3b574468fee4bd49f720b53ab0a0f94 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c3b574468fee4bd49f720b53ab0a0f94 2023-04-25 06:53:52,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405692658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,668 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c3b574468fee4bd49f720b53ab0a0f94, entries=150, sequenceid=105, filesize=11.6 K 2023-04-25 06:53:52,682 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5b0d5709c8a2455f93961781c91c6421 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5b0d5709c8a2455f93961781c91c6421 2023-04-25 06:53:52,694 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5b0d5709c8a2455f93961781c91c6421, entries=150, sequenceid=105, filesize=11.6 K 2023-04-25 06:53:52,706 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/12d68387f9714cdc9ee518abbf1b70b4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/12d68387f9714cdc9ee518abbf1b70b4 2023-04-25 06:53:52,724 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/12d68387f9714cdc9ee518abbf1b70b4, entries=150, sequenceid=105, filesize=11.6 K 2023-04-25 06:53:52,727 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 266734ad02022a1c1fca6ab96809adb8 in 636ms, sequenceid=105, compaction requested=true 2023-04-25 06:53:52,728 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:52,728 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:52,732 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,732 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:52,733 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,733 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412d1107[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:52,737 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:52,737 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,739 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:52,739 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,739 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:52,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:52,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:52,746 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:52,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:52,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:52,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:52,746 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:52,748 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:52,748 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,748 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:52,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,749 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,751 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,755 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:52,756 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:52,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:52,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:52,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:52,756 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:52,756 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,756 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,756 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:52,756 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:52,756 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,757 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,757 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,757 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,757 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:52,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:52,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:53:52,857 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:52,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:52,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:52,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:52,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:52,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:52,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:52,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:52,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:52,867 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:52,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:52,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:52,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:52,868 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,868 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:52,869 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:52,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:52,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:52,884 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:52,884 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:52,894 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:52,894 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:52,895 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:52,895 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:52,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:52,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405692887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:52,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:52,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65dcfd90[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:52,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:52,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:52,904 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:52,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:52,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405692894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405692895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:52,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405692897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405692900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:52,935 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:52,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:52,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:52,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:52,935 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:52,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:52,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/631def4486754da39c6b93479c98ea66 2023-04-25 06:53:52,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:52,942 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,945 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:52,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:52,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:52,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:52,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:52,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:52,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:52,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:52,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:52,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:52,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:52,954 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:52,954 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:52,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:52,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:53,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/0463cee3c5554a8fa0415fde23b321d3 2023-04-25 06:53:53,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405693002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405693010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405693011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405693011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405693012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,055 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:53,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:53,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:53,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:53,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:53,092 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:53,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:53,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:53,093 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,093 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,095 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,095 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:53,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:53,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:53,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:53,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:53,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:53,106 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:53,106 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:53,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:53,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:53,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:53,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/72a4fdf151e84475818b91506af8ddff 2023-04-25 06:53:53,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,116 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:53,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@232539fa[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:53,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:53,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,122 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,123 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:53,123 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:53,123 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:53,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:53,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,125 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:53,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:53,130 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:53,130 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:53,130 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:53,130 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:53,130 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:53,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/631def4486754da39c6b93479c98ea66 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/631def4486754da39c6b93479c98ea66 2023-04-25 06:53:53,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:53,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,140 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,140 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:53,141 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:53,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/631def4486754da39c6b93479c98ea66, entries=150, sequenceid=120, filesize=11.6 K 2023-04-25 06:53:53,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/0463cee3c5554a8fa0415fde23b321d3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0463cee3c5554a8fa0415fde23b321d3 2023-04-25 06:53:53,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0463cee3c5554a8fa0415fde23b321d3, entries=150, sequenceid=120, filesize=11.6 K 2023-04-25 06:53:53,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/72a4fdf151e84475818b91506af8ddff as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/72a4fdf151e84475818b91506af8ddff 2023-04-25 06:53:53,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/72a4fdf151e84475818b91506af8ddff, entries=150, sequenceid=120, filesize=11.6 K 2023-04-25 06:53:53,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 266734ad02022a1c1fca6ab96809adb8 in 341ms, sequenceid=120, compaction requested=true 2023-04-25 06:53:53,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:53,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:53,197 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:53:53,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:53,197 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:53:53,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:53,200 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:53:53,200 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:53:53,200 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,200 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/df9a9ba8f1db411a934437e82e1032e1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/dacd8fee0aca4d2db8744b1bed8cf932, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c3b574468fee4bd49f720b53ab0a0f94, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/631def4486754da39c6b93479c98ea66] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=46.5 K 2023-04-25 06:53:53,201 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:53:53,201 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:53:53,201 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,201 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0be88638532a47d4ad4d2c4b1b72a652, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/2aa4ac39663f47c080d0a0b3eb7a4d72, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5b0d5709c8a2455f93961781c91c6421, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0463cee3c5554a8fa0415fde23b321d3] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=46.5 K 2023-04-25 06:53:53,202 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting df9a9ba8f1db411a934437e82e1032e1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682405629968 2023-04-25 06:53:53,202 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 0be88638532a47d4ad4d2c4b1b72a652, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682405629968 2023-04-25 06:53:53,206 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting dacd8fee0aca4d2db8744b1bed8cf932, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682405630176 2023-04-25 06:53:53,208 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c3b574468fee4bd49f720b53ab0a0f94, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682405631346 2023-04-25 06:53:53,209 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2aa4ac39663f47c080d0a0b3eb7a4d72, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682405630176 2023-04-25 06:53:53,209 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 631def4486754da39c6b93479c98ea66, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682405632532 2023-04-25 06:53:53,210 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5b0d5709c8a2455f93961781c91c6421, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682405631346 2023-04-25 06:53:53,212 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 0463cee3c5554a8fa0415fde23b321d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682405632532 2023-04-25 06:53:53,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:53:53,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:53,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:53,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:53,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:53,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:53,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:53,248 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#22 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:53,248 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#23 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:53,250 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:53,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:53,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:53,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405693246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405693248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405693255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:53,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:53,277 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:53,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:53,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:53,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405693262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,292 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:53,299 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405693278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:53,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:53,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:53,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:53,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,364 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8d98f9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:53,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:53,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405693365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405693365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:53,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405693370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,374 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,375 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:53,375 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:53,375 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:53,375 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:53,375 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:53,375 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:53,375 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:53,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,378 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,383 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:53,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:53,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:53,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405693392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405693402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:53,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:53,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:53,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:53,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:53,504 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:53,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:53,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:53,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:53,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:53,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:53,512 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,512 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:53,512 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:53,512 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:53,515 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:53,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47810906[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:53,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:53,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:53,523 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:53,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:53,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:53,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:53,523 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:53,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,528 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:53,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:53,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:53,531 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:53,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:53,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405693572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405693576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405693582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405693597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405693606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:53,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:53,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:53,641 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:53,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:53,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:53,642 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,642 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,649 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:53,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:53,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:53,651 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,651 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:53,651 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:53,651 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:53,661 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:53,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b34c1b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:53,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:53,665 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,666 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:53,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:53,672 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:53,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:53,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:53,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:53,672 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:53,674 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,674 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,674 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:53,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:53,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:53,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,689 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:53,689 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:53,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/20512ec237e04b819f634f3cdb23f7a5 2023-04-25 06:53:53,786 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/9069e3264dee4cf8b77f5df26daeedb2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9069e3264dee4cf8b77f5df26daeedb2 2023-04-25 06:53:53,789 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:53,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:53,793 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/e01dfb3d2ec342e3a1426a282c9c1810 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/e01dfb3d2ec342e3a1426a282c9c1810 2023-04-25 06:53:53,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/535fdcfc5cbd4018ae0dcb5dedee4d76 2023-04-25 06:53:53,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:53,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:53,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:53,803 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:53,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:53,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:53,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,806 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:53,810 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into 9069e3264dee4cf8b77f5df26daeedb2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:53,810 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:53,810 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=12, startTime=1682405633197; duration=0sec 2023-04-25 06:53:53,810 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:53,810 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:53:53,813 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into e01dfb3d2ec342e3a1426a282c9c1810(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:53,813 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:53,813 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=12, startTime=1682405633197; duration=0sec 2023-04-25 06:53:53,813 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:53,813 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:53:53,813 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:53:53,813 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,813 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/31f1c032f6bf403995f5bedb7c7b6568, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/747633842a354cdd91f152e0d2794aab, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/12d68387f9714cdc9ee518abbf1b70b4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/72a4fdf151e84475818b91506af8ddff] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=46.5 K 2023-04-25 06:53:53,814 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 31f1c032f6bf403995f5bedb7c7b6568, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682405629968 2023-04-25 06:53:53,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:53,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:53,816 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 747633842a354cdd91f152e0d2794aab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682405630176 2023-04-25 06:53:53,816 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,816 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:53,816 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:53,816 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:53,817 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 12d68387f9714cdc9ee518abbf1b70b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682405631346 2023-04-25 06:53:53,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,819 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 72a4fdf151e84475818b91506af8ddff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682405632532 2023-04-25 06:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f5347be[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:53,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:53,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:53,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:53,829 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:53,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/aea8a11944714b169c85008da5ecb96b 2023-04-25 06:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,840 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#26 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:53,844 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:53,848 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:53,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:53,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:53,849 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:53,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/20512ec237e04b819f634f3cdb23f7a5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/20512ec237e04b819f634f3cdb23f7a5 2023-04-25 06:53:53,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,862 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,865 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,865 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,865 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,866 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,866 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:53,866 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:53,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:53,866 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,866 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,866 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,866 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:53,867 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:53,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/20512ec237e04b819f634f3cdb23f7a5, entries=200, sequenceid=138, filesize=14.1 K 2023-04-25 06:53:53,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/535fdcfc5cbd4018ae0dcb5dedee4d76 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/535fdcfc5cbd4018ae0dcb5dedee4d76 2023-04-25 06:53:53,874 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/500f6f4cfb574704a54a5af8f966125f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/500f6f4cfb574704a54a5af8f966125f 2023-04-25 06:53:53,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405693878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405693878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/535fdcfc5cbd4018ae0dcb5dedee4d76, entries=150, sequenceid=138, filesize=11.8 K 2023-04-25 06:53:53,890 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into 500f6f4cfb574704a54a5af8f966125f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:53,890 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:53,890 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=12, startTime=1682405633197; duration=0sec 2023-04-25 06:53:53,890 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:53,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/aea8a11944714b169c85008da5ecb96b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/aea8a11944714b169c85008da5ecb96b 2023-04-25 06:53:53,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405693890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/aea8a11944714b169c85008da5ecb96b, entries=150, sequenceid=138, filesize=11.8 K 2023-04-25 06:53:53,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 266734ad02022a1c1fca6ab96809adb8 in 691ms, sequenceid=138, compaction requested=false 2023-04-25 06:53:53,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:53,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:53,913 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:53:53,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:53,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:53,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:53,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:53,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:53,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:53,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405693957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:53,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405693961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,967 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:53,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:53,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:53,972 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:53,972 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:53,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:53,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,974 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:53,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:53,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:53,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:53,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:53,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:53,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:53,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:53,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:53,981 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d9b54fd[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:53,985 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:53,985 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,986 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:53,987 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,988 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:53,990 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:53,994 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:53,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:53,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:53,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:53,994 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:53,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:53,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:53,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:53,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,001 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,005 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:54,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,005 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:54,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,005 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,005 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:54,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405694062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405694068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,106 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:54,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:54,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:54,125 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:54,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:54,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:54,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,128 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:54,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:54,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:54,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:54,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:54,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:54,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:54,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:54,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:54,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21dbd71[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:54,170 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:54,170 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:54,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:54,179 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:54,183 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:54,184 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:54,184 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:54,184 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:54,184 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:54,186 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,186 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,186 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,190 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,193 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:54,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:54,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,196 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:54,196 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405694267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405694274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,296 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:54,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:54,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:54,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:54,308 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:54,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:54,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:54,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,315 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,315 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:54,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:54,320 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,320 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:54,320 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:54,320 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:54,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,324 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,324 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:54,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d11eabd[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:54,327 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:54,327 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,328 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,329 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:54,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:54,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:54,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:54,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:54,338 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:54,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:54,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:54,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:54,338 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:54,341 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,341 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,343 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,349 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:54,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:54,353 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:54,353 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:54,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:54,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/2c5ccc4957674ea1bec156476b0d3572 2023-04-25 06:53:54,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405694384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405694389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405694403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/46aa745efeab4b429f19cf1e4e9d8280 2023-04-25 06:53:54,453 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:54,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:54,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:54,464 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:54,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:54,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:54,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,466 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:54,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:54,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:54,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:54,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:54,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,478 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8aa3816[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:54,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:54,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:54,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:54,486 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:54,489 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:54,489 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:54,489 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:54,489 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:54,489 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:54,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,493 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:54,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,502 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,502 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,502 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:54,502 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:54,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405694577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:54,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405694577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,602 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:54,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:54,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:54,607 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:54,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:54,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:54,608 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,608 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,610 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:54,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:54,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:54,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:54,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:54,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:54,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:54,622 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,622 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f81a942[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:54,626 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:54,626 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,628 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:54,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:54,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,633 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,634 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,634 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:54,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:54,637 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:54,637 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:54,637 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:54,637 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:54,637 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:54,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,642 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,643 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:54,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:54,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,648 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:54,648 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:54,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,748 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:54,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:54,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:54,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:54,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:54,754 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:54,754 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,754 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:54,754 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:54,754 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,754 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,756 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,756 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,757 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,757 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:54,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:54,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:54,759 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,759 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:54,759 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:54,759 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:54,760 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:54,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@701879f6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:54,767 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:54,767 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,768 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:54,768 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,768 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:54,771 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:54,774 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:54,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:54,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:54,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:54,774 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:54,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,783 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:54,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:54,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,784 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:54,784 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:54,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/1cc47876b2974ac282447b901dcefc09 2023-04-25 06:53:54,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/2c5ccc4957674ea1bec156476b0d3572 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2c5ccc4957674ea1bec156476b0d3572 2023-04-25 06:53:54,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2c5ccc4957674ea1bec156476b0d3572, entries=200, sequenceid=157, filesize=14.1 K 2023-04-25 06:53:54,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/46aa745efeab4b429f19cf1e4e9d8280 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/46aa745efeab4b429f19cf1e4e9d8280 2023-04-25 06:53:54,884 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:54,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/46aa745efeab4b429f19cf1e4e9d8280, entries=150, sequenceid=157, filesize=11.8 K 2023-04-25 06:53:54,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:54,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/1cc47876b2974ac282447b901dcefc09 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1cc47876b2974ac282447b901dcefc09 2023-04-25 06:53:54,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:54,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:54,899 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:54,899 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,899 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:54,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:54,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1cc47876b2974ac282447b901dcefc09, entries=150, sequenceid=157, filesize=11.8 K 2023-04-25 06:53:54,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:54,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:54,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:54,917 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,917 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:54,918 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:54,918 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,918 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 266734ad02022a1c1fca6ab96809adb8 in 1005ms, sequenceid=157, compaction requested=true 2023-04-25 06:53:54,918 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:54,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:54,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:54,922 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:54,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:54,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:54,924 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:54,924 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:53:54,925 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,925 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9069e3264dee4cf8b77f5df26daeedb2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/535fdcfc5cbd4018ae0dcb5dedee4d76, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/46aa745efeab4b429f19cf1e4e9d8280] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=35.3 K 2023-04-25 06:53:54,925 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,926 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:54,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:54,926 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9069e3264dee4cf8b77f5df26daeedb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682405632532 2023-04-25 06:53:54,928 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 535fdcfc5cbd4018ae0dcb5dedee4d76, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1682405632891 2023-04-25 06:53:54,928 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40978 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:54,928 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:53:54,928 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:54,929 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/e01dfb3d2ec342e3a1426a282c9c1810, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/20512ec237e04b819f634f3cdb23f7a5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2c5ccc4957674ea1bec156476b0d3572] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=40.0 K 2023-04-25 06:53:54,929 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 46aa745efeab4b429f19cf1e4e9d8280, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405633252 2023-04-25 06:53:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,934 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting e01dfb3d2ec342e3a1426a282c9c1810, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682405632532 2023-04-25 06:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,935 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 20512ec237e04b819f634f3cdb23f7a5, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1682405632874 2023-04-25 06:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,937 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 2c5ccc4957674ea1bec156476b0d3572, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405633252 2023-04-25 06:53:54,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:54,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228e214a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:54,937 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:54,937 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:54,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:54,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:54,941 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:54,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:54,949 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:54,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:54,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:54,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:54,949 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:54,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:54,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:54,953 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#30 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,956 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,962 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:54,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:54,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:54,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:54,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:54,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:54,963 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:54,963 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:54,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:54,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:54,990 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/e3ba237709c647c69d8e423554ae38e3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e3ba237709c647c69d8e423554ae38e3 2023-04-25 06:53:54,998 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into e3ba237709c647c69d8e423554ae38e3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:54,998 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:54,998 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=13, startTime=1682405634922; duration=0sec 2023-04-25 06:53:54,998 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:54,998 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:55,001 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:55,001 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:53:55,001 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:55,001 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/500f6f4cfb574704a54a5af8f966125f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/aea8a11944714b169c85008da5ecb96b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1cc47876b2974ac282447b901dcefc09] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=35.3 K 2023-04-25 06:53:55,002 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 500f6f4cfb574704a54a5af8f966125f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682405632532 2023-04-25 06:53:55,002 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting aea8a11944714b169c85008da5ecb96b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1682405632891 2023-04-25 06:53:55,004 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1cc47876b2974ac282447b901dcefc09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405633252 2023-04-25 06:53:55,014 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#32 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:55,029 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/4b0f25723ed747dcbcf45d3a345c2e2a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4b0f25723ed747dcbcf45d3a345c2e2a 2023-04-25 06:53:55,040 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into 4b0f25723ed747dcbcf45d3a345c2e2a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:55,040 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:55,040 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=13, startTime=1682405634922; duration=0sec 2023-04-25 06:53:55,040 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:55,063 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:55,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:55,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:55,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:55,071 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:55,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:55,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:55,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:55,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:55,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:55,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:55,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:55,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:55,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:55,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:55,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:55,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:55,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:55,082 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:55,082 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:55,082 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:53:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:55,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:55,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:55,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405695111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405695113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405695215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405695218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405695391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,395 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/5b0eda36652e48b284c82bedb32fa8ed as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5b0eda36652e48b284c82bedb32fa8ed 2023-04-25 06:53:55,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405695399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,416 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into 5b0eda36652e48b284c82bedb32fa8ed(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:55,416 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:55,416 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=13, startTime=1682405634919; duration=0sec 2023-04-25 06:53:55,416 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:55,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405695415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405695419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405695422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,518 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/a3839869210542edaa1ca8d9c56e7f6d 2023-04-25 06:53:55,547 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/9cfa964eeada4b6596c777d82c2f5424 2023-04-25 06:53:55,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405695742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:55,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405695743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:55,968 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/3498065cc3e540eab2e5ad16c9f3797f 2023-04-25 06:53:55,980 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/a3839869210542edaa1ca8d9c56e7f6d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/a3839869210542edaa1ca8d9c56e7f6d 2023-04-25 06:53:55,988 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/a3839869210542edaa1ca8d9c56e7f6d, entries=150, sequenceid=175, filesize=11.8 K 2023-04-25 06:53:55,992 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/9cfa964eeada4b6596c777d82c2f5424 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9cfa964eeada4b6596c777d82c2f5424 2023-04-25 06:53:55,999 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9cfa964eeada4b6596c777d82c2f5424, entries=150, sequenceid=175, filesize=11.8 K 2023-04-25 06:53:56,000 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/3498065cc3e540eab2e5ad16c9f3797f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3498065cc3e540eab2e5ad16c9f3797f 2023-04-25 06:53:56,006 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3498065cc3e540eab2e5ad16c9f3797f, entries=150, sequenceid=175, filesize=11.8 K 2023-04-25 06:53:56,007 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 266734ad02022a1c1fca6ab96809adb8 in 925ms, sequenceid=175, compaction requested=false 2023-04-25 06:53:56,008 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:56,008 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:56,008 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:56,009 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:56,009 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,009 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:56,009 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:56,011 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,011 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:56,011 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,011 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:56,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:56,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3dc785[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:56,013 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:56,013 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,014 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,014 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:56,015 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:56,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:56,015 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:56,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:56,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:56,022 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:56,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:56,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:56,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:56,022 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:56,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:56,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:56,025 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:56,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:56,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:56,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:56,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:56,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:56,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:56,032 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:56,033 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:56,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:56,133 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:56,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:56,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:56,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:56,137 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:56,137 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:56,137 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:56,137 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:56,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,139 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:56,139 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:56,140 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,140 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:56,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:56,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:56,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:56,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:56,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:56,144 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:56,144 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:56,144 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:56,144 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:56,144 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:53:56,144 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:56,145 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:56,145 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:56,145 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:56,145 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:56,145 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:56,164 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd 2023-04-25 06:53:56,185 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/58d759608e354b4db9de38e259ff337e 2023-04-25 06:53:56,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:56,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:56,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405696272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405696276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405696378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405696381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405696583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405696586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,604 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/9ff4edb3f91541569b7f87f1b6fb3406 2023-04-25 06:53:56,613 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd 2023-04-25 06:53:56,622 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd, entries=150, sequenceid=192, filesize=11.8 K 2023-04-25 06:53:56,624 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/58d759608e354b4db9de38e259ff337e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/58d759608e354b4db9de38e259ff337e 2023-04-25 06:53:56,633 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/58d759608e354b4db9de38e259ff337e, entries=150, sequenceid=192, filesize=11.8 K 2023-04-25 06:53:56,635 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/9ff4edb3f91541569b7f87f1b6fb3406 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9ff4edb3f91541569b7f87f1b6fb3406 2023-04-25 06:53:56,641 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9ff4edb3f91541569b7f87f1b6fb3406, entries=150, sequenceid=192, filesize=11.8 K 2023-04-25 06:53:56,643 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 266734ad02022a1c1fca6ab96809adb8 in 499ms, sequenceid=192, compaction requested=true 2023-04-25 06:53:56,643 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:56,643 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:56,650 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:56,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:56,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6964aa19[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:56,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:56,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:56,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:56,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:56,661 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:56,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:56,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:56,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:56,661 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:56,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:56,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:56,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:56,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:56,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:56,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:56,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:56,670 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:56,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,671 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:56,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:56,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:56,671 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:56,671 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:56,772 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:56,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:56,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:56,777 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:56,777 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:56,778 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:56,778 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:56,778 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,778 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,780 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:56,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:56,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:56,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:56,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:56,788 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:56,788 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:56,788 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:53:56,788 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:56,789 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:56,789 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:56,789 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:56,789 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:56,789 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:56,814 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/94a90d34011c488fae1dd7810ad7b9c4 2023-04-25 06:53:56,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:56,895 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:56,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405696927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:56,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:56,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405696933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405697035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405697041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,241 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5381b5fa70f54cc39902bbf0b75d353e 2023-04-25 06:53:57,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405697239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405697251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405697412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,417 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:53:57,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405697420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,425 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:53:57,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405697444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,448 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4199 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:53:57,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43142 deadline: 1682405697543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:57,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682405697557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,693 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/016eb488528a4d929d7bf9f37f54ba97 2023-04-25 06:53:57,704 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/94a90d34011c488fae1dd7810ad7b9c4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/94a90d34011c488fae1dd7810ad7b9c4 2023-04-25 06:53:57,713 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/94a90d34011c488fae1dd7810ad7b9c4, entries=150, sequenceid=208, filesize=11.8 K 2023-04-25 06:53:57,715 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5381b5fa70f54cc39902bbf0b75d353e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5381b5fa70f54cc39902bbf0b75d353e 2023-04-25 06:53:57,729 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5381b5fa70f54cc39902bbf0b75d353e, entries=150, sequenceid=208, filesize=11.8 K 2023-04-25 06:53:57,731 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/016eb488528a4d929d7bf9f37f54ba97 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/016eb488528a4d929d7bf9f37f54ba97 2023-04-25 06:53:57,741 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/016eb488528a4d929d7bf9f37f54ba97, entries=150, sequenceid=208, filesize=11.8 K 2023-04-25 06:53:57,749 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 266734ad02022a1c1fca6ab96809adb8 in 961ms, sequenceid=208, compaction requested=true 2023-04-25 06:53:57,749 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:57,749 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:57,749 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a80b90a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:57,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:57,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,868 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:57,869 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,869 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:57,914 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:57,919 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:57,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:57,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:57,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:57,919 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:57,937 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:57,937 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:57,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:57,937 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:57,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:57,938 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,940 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:57,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:57,976 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:57,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:57,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:57,977 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:58,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:53:58,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:58,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:58,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:58,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:58,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:58,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:58,077 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:58,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:58,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:58,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:58,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:58,084 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:58,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:58,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:58,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405698086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405698092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,153 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,153 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,153 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,153 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:58,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:58,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:58,156 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,156 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:58,157 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:58,157 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:58,191 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,192 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,192 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:58,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9574761[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:58,194 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:58,194 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405698193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405698198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:58,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:58,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,306 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:58,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:58,310 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:58,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:58,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:58,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:58,310 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:58,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,357 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405698397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405698403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/4d0e73e1081843c8a3917bde4d8453b3 2023-04-25 06:53:58,483 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,483 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,483 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,483 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:58,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:58,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:58,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,484 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:58,485 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:58,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5bdadaba092042968b6d1f6150de55c2 2023-04-25 06:53:58,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:58,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:58,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:58,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:58,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:58,592 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:58,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:58,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:58,593 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,593 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,595 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,596 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,596 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:58,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:58,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:58,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:58,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:58,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:58,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:58,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:58,609 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c43a821[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:58,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:58,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,613 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:58,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:58,616 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:58,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:58,620 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:58,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:58,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:58,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:58,620 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:58,624 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,624 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,624 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,626 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:58,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:58,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,632 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:58,632 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:58,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:58,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405698702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:58,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405698705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,732 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:58,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:58,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:58,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:58,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:58,744 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:58,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:58,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:58,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,746 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,747 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,748 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,748 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:58,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:58,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:58,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:58,772 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,772 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:58,772 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:58,772 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:58,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:58,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@503bfdd2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:58,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:58,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,780 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:58,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:58,782 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,783 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:58,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:58,787 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:58,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:58,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:58,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:58,787 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:58,789 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,789 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,789 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,798 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,803 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,803 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:58,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:58,803 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,804 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:58,804 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:58,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,904 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:58,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:58,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:58,919 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:58,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:58,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:58,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,922 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:58,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:58,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:58,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:58,932 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,932 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:58,932 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:58,932 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:58,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,934 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:58,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f2c5636[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:58,937 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:58,937 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,941 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/2b574aaa22d9488483742ccb252b4c4e 2023-04-25 06:53:58,941 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:58,942 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,942 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:58,944 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:58,948 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:58,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:58,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:58,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:58,948 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:58,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/4d0e73e1081843c8a3917bde4d8453b3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4d0e73e1081843c8a3917bde4d8453b3 2023-04-25 06:53:58,951 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,951 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:58,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,951 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4d0e73e1081843c8a3917bde4d8453b3, entries=200, sequenceid=225, filesize=14.1 K 2023-04-25 06:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5bdadaba092042968b6d1f6150de55c2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bdadaba092042968b6d1f6150de55c2 2023-04-25 06:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:58,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:58,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:58,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:58,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:58,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:58,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:58,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:58,966 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:58,966 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:58,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bdadaba092042968b6d1f6150de55c2, entries=150, sequenceid=225, filesize=11.8 K 2023-04-25 06:53:58,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/2b574aaa22d9488483742ccb252b4c4e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/2b574aaa22d9488483742ccb252b4c4e 2023-04-25 06:53:58,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/2b574aaa22d9488483742ccb252b4c4e, entries=150, sequenceid=225, filesize=11.8 K 2023-04-25 06:53:58,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 266734ad02022a1c1fca6ab96809adb8 in 935ms, sequenceid=225, compaction requested=true 2023-04-25 06:53:58,986 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:58,987 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:53:58,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:58,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:58,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:53:58,988 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:53:58,990 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62912 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:53:58,990 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:53:58,990 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:58,990 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5b0eda36652e48b284c82bedb32fa8ed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/a3839869210542edaa1ca8d9c56e7f6d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/94a90d34011c488fae1dd7810ad7b9c4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4d0e73e1081843c8a3917bde4d8453b3] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=61.4 K 2023-04-25 06:53:58,991 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5b0eda36652e48b284c82bedb32fa8ed, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405633252 2023-04-25 06:53:58,992 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:53:59,001 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:53:59,001 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,002 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e3ba237709c647c69d8e423554ae38e3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9cfa964eeada4b6596c777d82c2f5424, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/58d759608e354b4db9de38e259ff337e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5381b5fa70f54cc39902bbf0b75d353e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bdadaba092042968b6d1f6150de55c2] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=59.1 K 2023-04-25 06:53:59,002 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a3839869210542edaa1ca8d9c56e7f6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1682405633945 2023-04-25 06:53:59,004 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting e3ba237709c647c69d8e423554ae38e3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405633252 2023-04-25 06:53:59,005 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3b18bf8ab4704b3cb4ed2b6edb04d2bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682405635107 2023-04-25 06:53:59,005 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9cfa964eeada4b6596c777d82c2f5424, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1682405633945 2023-04-25 06:53:59,006 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 94a90d34011c488fae1dd7810ad7b9c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1682405636262 2023-04-25 06:53:59,006 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 58d759608e354b4db9de38e259ff337e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682405635107 2023-04-25 06:53:59,007 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4d0e73e1081843c8a3917bde4d8453b3, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682405636918 2023-04-25 06:53:59,007 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5381b5fa70f54cc39902bbf0b75d353e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1682405636262 2023-04-25 06:53:59,008 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5bdadaba092042968b6d1f6150de55c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682405636925 2023-04-25 06:53:59,030 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:59,038 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#46 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:59,067 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:59,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:59,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:59,075 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:59,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:59,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:59,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:59,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:59,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:59,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:59,090 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/9accb2d718e34f80a9173c2ee387324b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9accb2d718e34f80a9173c2ee387324b 2023-04-25 06:53:59,091 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/4631af755f30449c8361bf95bb00495b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/4631af755f30449c8361bf95bb00495b 2023-04-25 06:53:59,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:59,098 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:59,098 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:59,098 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,098 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:59,098 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:53:59,098 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:59,101 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into 9accb2d718e34f80a9173c2ee387324b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:59,102 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,102 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:59,107 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,108 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:59,108 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,102 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:59,108 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=11, startTime=1682405638987; duration=0sec 2023-04-25 06:53:59,109 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:59,109 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:53:59,110 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into 4631af755f30449c8361bf95bb00495b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:59,110 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:59,110 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=11, startTime=1682405638987; duration=0sec 2023-04-25 06:53:59,110 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:59,114 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:53:59,114 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:53:59,114 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,114 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4b0f25723ed747dcbcf45d3a345c2e2a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3498065cc3e540eab2e5ad16c9f3797f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9ff4edb3f91541569b7f87f1b6fb3406, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/016eb488528a4d929d7bf9f37f54ba97, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/2b574aaa22d9488483742ccb252b4c4e] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=59.1 K 2023-04-25 06:53:59,115 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4b0f25723ed747dcbcf45d3a345c2e2a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405633252 2023-04-25 06:53:59,115 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3498065cc3e540eab2e5ad16c9f3797f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1682405633945 2023-04-25 06:53:59,116 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9ff4edb3f91541569b7f87f1b6fb3406, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682405635107 2023-04-25 06:53:59,116 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 016eb488528a4d929d7bf9f37f54ba97, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1682405636262 2023-04-25 06:53:59,117 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2b574aaa22d9488483742ccb252b4c4e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682405636925 2023-04-25 06:53:59,142 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:59,150 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/2af9708f3169496780ba99dec6d205d0 2023-04-25 06:53:59,210 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/81358bc1036041139e3b42d8fd0c92c9 2023-04-25 06:53:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:59,214 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405699252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405699256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,261 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/7ff60f9c0b95470ab6d6fd498d7584f7 2023-04-25 06:53:59,271 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/2af9708f3169496780ba99dec6d205d0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2af9708f3169496780ba99dec6d205d0 2023-04-25 06:53:59,280 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2af9708f3169496780ba99dec6d205d0, entries=150, sequenceid=240, filesize=11.8 K 2023-04-25 06:53:59,281 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/81358bc1036041139e3b42d8fd0c92c9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/81358bc1036041139e3b42d8fd0c92c9 2023-04-25 06:53:59,291 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/81358bc1036041139e3b42d8fd0c92c9, entries=150, sequenceid=240, filesize=11.8 K 2023-04-25 06:53:59,292 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/7ff60f9c0b95470ab6d6fd498d7584f7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/7ff60f9c0b95470ab6d6fd498d7584f7 2023-04-25 06:53:59,303 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/7ff60f9c0b95470ab6d6fd498d7584f7, entries=150, sequenceid=240, filesize=11.8 K 2023-04-25 06:53:59,307 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 266734ad02022a1c1fca6ab96809adb8 in 209ms, sequenceid=240, compaction requested=false 2023-04-25 06:53:59,307 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:59,308 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,308 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:59,315 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf374fd[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:59,320 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:59,320 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,321 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:59,322 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,322 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:59,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:59,328 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:59,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:59,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:59,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:59,328 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:59,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:59,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,344 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:59,344 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:59,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:59,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:59,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:59,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:59,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:59,345 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:59,345 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:59,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:53:59,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:59,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:53:59,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:59,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:59,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:59,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405699393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405699394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/d5a8cec25a1b433e8107d7b1727f09e3 2023-04-25 06:53:59,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a 2023-04-25 06:53:59,445 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:59,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:59,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:59,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:59,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:59,464 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:59,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:59,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:59,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,466 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:59,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,468 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,468 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:59,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:59,482 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,482 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:59,482 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:59,482 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:59,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/5c403171c1cc4c14a8ec69c3ce762446 2023-04-25 06:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,488 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:59,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3df0c1e2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:59,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:59,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,497 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:59,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:59,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:59,498 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,498 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:59,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/d5a8cec25a1b433e8107d7b1727f09e3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/d5a8cec25a1b433e8107d7b1727f09e3 2023-04-25 06:53:59,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:59,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:59,504 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:59,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:59,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:59,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:59,504 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:59,506 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,506 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,506 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:59,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405699505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405699505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/d5a8cec25a1b433e8107d7b1727f09e3, entries=150, sequenceid=261, filesize=11.9 K 2023-04-25 06:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,511 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a 2023-04-25 06:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:59,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:59,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a, entries=150, sequenceid=261, filesize=11.9 K 2023-04-25 06:53:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:59,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,521 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,522 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:59,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:59,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:59,522 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,522 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,522 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:59,522 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:59,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:59,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/5c403171c1cc4c14a8ec69c3ce762446 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/5c403171c1cc4c14a8ec69c3ce762446 2023-04-25 06:53:59,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/5c403171c1cc4c14a8ec69c3ce762446, entries=150, sequenceid=261, filesize=11.9 K 2023-04-25 06:53:59,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 266734ad02022a1c1fca6ab96809adb8 in 170ms, sequenceid=261, compaction requested=true 2023-04-25 06:53:59,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:59,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:53:59,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:59,532 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:59,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:53:59,534 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36720 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:59,534 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:53:59,534 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,534 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9accb2d718e34f80a9173c2ee387324b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2af9708f3169496780ba99dec6d205d0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/d5a8cec25a1b433e8107d7b1727f09e3] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=35.9 K 2023-04-25 06:53:59,535 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9accb2d718e34f80a9173c2ee387324b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682405636925 2023-04-25 06:53:59,535 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 2af9708f3169496780ba99dec6d205d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682405638083 2023-04-25 06:53:59,536 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d5a8cec25a1b433e8107d7b1727f09e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1682405639248 2023-04-25 06:53:59,546 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:59,600 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/6cbdbd8b091e4c48b9fed0e388daca66 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/6cbdbd8b091e4c48b9fed0e388daca66 2023-04-25 06:53:59,611 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into 6cbdbd8b091e4c48b9fed0e388daca66(size=12.3 K), total size for store is 35.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:59,611 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:59,611 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=11, startTime=1682405638987; duration=0sec 2023-04-25 06:53:59,611 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:53:59,611 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:53:59,615 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36720 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:53:59,615 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:53:59,615 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,615 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/4631af755f30449c8361bf95bb00495b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/81358bc1036041139e3b42d8fd0c92c9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=35.9 K 2023-04-25 06:53:59,616 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4631af755f30449c8361bf95bb00495b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682405636925 2023-04-25 06:53:59,616 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 81358bc1036041139e3b42d8fd0c92c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682405638083 2023-04-25 06:53:59,617 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting d7b42f5a9d1a41f0ba243a2b5de6ff1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1682405639248 2023-04-25 06:53:59,622 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:59,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:59,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:59,640 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:59,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:59,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:59,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,642 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:53:59,643 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:59,643 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:59,644 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,644 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:59,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:59,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:59,648 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:59,649 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:53:59,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:59,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:59,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,650 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:59,650 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,685 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/5e0335b5d14d40b19f021d50bb7b910d 2023-04-25 06:53:59,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:59,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:59,731 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/b4f9c071b0d749c19b3f41b7be4a79d9 2023-04-25 06:53:59,767 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/a93718d73c3e45b59e3c2e1812219a16 2023-04-25 06:53:59,776 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/5e0335b5d14d40b19f021d50bb7b910d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5e0335b5d14d40b19f021d50bb7b910d 2023-04-25 06:53:59,786 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5e0335b5d14d40b19f021d50bb7b910d, entries=150, sequenceid=275, filesize=11.9 K 2023-04-25 06:53:59,790 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/b4f9c071b0d749c19b3f41b7be4a79d9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/b4f9c071b0d749c19b3f41b7be4a79d9 2023-04-25 06:53:59,798 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/b4f9c071b0d749c19b3f41b7be4a79d9, entries=150, sequenceid=275, filesize=11.9 K 2023-04-25 06:53:59,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/a93718d73c3e45b59e3c2e1812219a16 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a93718d73c3e45b59e3c2e1812219a16 2023-04-25 06:53:59,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405699798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405699817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,823 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a93718d73c3e45b59e3c2e1812219a16, entries=150, sequenceid=275, filesize=11.9 K 2023-04-25 06:53:59,825 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 266734ad02022a1c1fca6ab96809adb8 in 176ms, sequenceid=275, compaction requested=true 2023-04-25 06:53:59,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:59,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:59,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32aa6ee3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:59,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:59,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,838 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:59,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:59,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:59,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:59,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:59,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:59,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,841 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:59,841 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:59,841 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:59,841 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:59,841 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:59,841 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:59,841 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:59,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,845 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:59,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:59,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:59,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:53:59,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:53:59,852 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:53:59,852 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:53:59,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:59,852 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,852 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:59,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:53:59,916 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:53:59,917 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:53:59,917 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,917 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:53:59,917 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,917 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:53:59,917 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:53:59,952 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:53:59,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:53:59,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:53:59,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405699954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:53:59,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405699960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:53:59,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:53:59,965 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:53:59,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:53:59,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:53:59,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,967 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:53:59,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,967 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:53:59,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:53:59,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:53:59,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:53:59,971 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,971 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:53:59,971 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:53:59,971 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:53:59,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,974 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,974 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,976 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/6be7818df43b42b18a1f86559f31d173 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6be7818df43b42b18a1f86559f31d173 2023-04-25 06:53:59,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5a1366[Count = 0] remaining members to acquire global barrier 2023-04-25 06:53:59,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:53:59,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,985 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into 6be7818df43b42b18a1f86559f31d173(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:53:59,985 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:53:59,985 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=13, startTime=1682405639532; duration=0sec 2023-04-25 06:53:59,985 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:53:59,985 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:53:59,985 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:53:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:53:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:53:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:53:59,986 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,986 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:53:59,987 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48906 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:53:59,987 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:53:59,987 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:53:59,987 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/6cbdbd8b091e4c48b9fed0e388daca66, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/7ff60f9c0b95470ab6d6fd498d7584f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/5c403171c1cc4c14a8ec69c3ce762446, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a93718d73c3e45b59e3c2e1812219a16] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=47.8 K 2023-04-25 06:53:59,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:53:59,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:53:59,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:53:59,988 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,988 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6cbdbd8b091e4c48b9fed0e388daca66, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682405636925 2023-04-25 06:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,989 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 7ff60f9c0b95470ab6d6fd498d7584f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682405638083 2023-04-25 06:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:53:59,990 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5c403171c1cc4c14a8ec69c3ce762446, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1682405639248 2023-04-25 06:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,990 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a93718d73c3e45b59e3c2e1812219a16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682405639371 2023-04-25 06:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:53:59,993 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:53:59,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:53:59,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:53:59,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:53:59,993 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:53:59,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:53:59,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:53:59,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:53:59,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:53:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:53:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:53:59,998 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:53:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,009 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,009 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,010 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,010 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,010 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,010 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,010 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,010 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,010 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,010 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:00,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:00,011 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:00,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:00,020 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:00,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405700061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,065 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/eeceed4923a6463fb09ebbf6839a4b6f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/eeceed4923a6463fb09ebbf6839a4b6f 2023-04-25 06:54:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405700065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,080 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into eeceed4923a6463fb09ebbf6839a4b6f(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:00,080 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:00,080 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=13, startTime=1682405639532; duration=0sec 2023-04-25 06:54:00,080 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:00,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:00,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:00,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:00,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:00,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:00,136 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:00,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:00,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:00,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,141 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,141 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:00,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:00,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:00,155 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,155 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:00,155 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:00,155 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:00,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:00,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:00,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7829d0ec[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:00,160 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:00,160 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,162 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:00,164 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,165 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:00,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:00,169 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:00,169 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:00,169 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:00,169 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:00,169 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:00,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,172 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,172 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,179 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,181 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:00,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:00,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:00,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:00,182 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,182 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405700266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405700271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:00,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:00,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:00,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:00,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:00,297 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:00,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:00,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:00,299 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,299 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,301 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:00,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:00,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:00,310 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,310 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:00,310 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:00,310 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:00,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43edf1da[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:00,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:00,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,322 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:00,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:00,325 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:00,328 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:00,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:00,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:00,329 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:00,329 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:00,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,339 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,339 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:00,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:00,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,347 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:00,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,347 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:00,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,357 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/049039c5ee9e47f2ba65ca6a7977035d 2023-04-25 06:54:00,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/daeea9611beb4ecba1a84feed985fd49 2023-04-25 06:54:00,447 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:00,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:00,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:00,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:00,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:00,460 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:00,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:00,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:00,461 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,461 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/b50aa104e77c49dba8ac93e30fc4a978 2023-04-25 06:54:00,464 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/d7687ccd70f04ab4823c4e22dd583e4d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d7687ccd70f04ab4823c4e22dd583e4d 2023-04-25 06:54:00,469 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,469 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,470 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,470 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:00,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/049039c5ee9e47f2ba65ca6a7977035d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/049039c5ee9e47f2ba65ca6a7977035d 2023-04-25 06:54:00,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:00,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:00,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:00,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:00,477 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,477 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:00,477 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:00,477 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:00,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:00,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:00,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:00,478 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into d7687ccd70f04ab4823c4e22dd583e4d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:00,478 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:00,478 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=12, startTime=1682405639532; duration=0sec 2023-04-25 06:54:00,478 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:00,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,483 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,483 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,483 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/049039c5ee9e47f2ba65ca6a7977035d, entries=200, sequenceid=295, filesize=14.3 K 2023-04-25 06:54:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/daeea9611beb4ecba1a84feed985fd49 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/daeea9611beb4ecba1a84feed985fd49 2023-04-25 06:54:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:00,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22515232[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:00,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:00,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,489 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:00,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:00,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:00,490 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,490 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:00,496 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:00,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/daeea9611beb4ecba1a84feed985fd49, entries=150, sequenceid=295, filesize=11.9 K 2023-04-25 06:54:00,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/b50aa104e77c49dba8ac93e30fc4a978 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/b50aa104e77c49dba8ac93e30fc4a978 2023-04-25 06:54:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:00,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:00,504 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:00,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:00,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:00,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:00,504 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:00,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/b50aa104e77c49dba8ac93e30fc4a978, entries=150, sequenceid=295, filesize=11.9 K 2023-04-25 06:54:00,510 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 266734ad02022a1c1fca6ab96809adb8 in 595ms, sequenceid=295, compaction requested=true 2023-04-25 06:54:00,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:00,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:00,512 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:00,512 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:00,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:00,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,516 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:00,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,516 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:54:00,517 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,517 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/eeceed4923a6463fb09ebbf6839a4b6f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/b4f9c071b0d749c19b3f41b7be4a79d9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/daeea9611beb4ecba1a84feed985fd49] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=36.3 K 2023-04-25 06:54:00,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:00,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,517 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting eeceed4923a6463fb09ebbf6839a4b6f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1682405639248 2023-04-25 06:54:00,517 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39562 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:00,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,518 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:54:00,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:00,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,518 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,518 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,518 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6be7818df43b42b18a1f86559f31d173, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5e0335b5d14d40b19f021d50bb7b910d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/049039c5ee9e47f2ba65ca6a7977035d] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=38.6 K 2023-04-25 06:54:00,518 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b4f9c071b0d749c19b3f41b7be4a79d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682405639371 2023-04-25 06:54:00,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:00,519 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6be7818df43b42b18a1f86559f31d173, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1682405639248 2023-04-25 06:54:00,518 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:00,519 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting daeea9611beb4ecba1a84feed985fd49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682405639795 2023-04-25 06:54:00,519 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:00,519 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5e0335b5d14d40b19f021d50bb7b910d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682405639371 2023-04-25 06:54:00,521 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 049039c5ee9e47f2ba65ca6a7977035d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682405639741 2023-04-25 06:54:00,538 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:00,541 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#64 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:00,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:54:00,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:00,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:00,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:00,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:00,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:00,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:00,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405700611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405700613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,619 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:00,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:00,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:00,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:00,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:00,632 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:00,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:00,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:00,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,634 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,636 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,636 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:00,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:00,637 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,638 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:00,638 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:00,638 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,641 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d1fb43d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:00,643 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:00,643 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,645 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:00,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:00,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:00,646 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:00,649 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:00,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:00,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:00,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:00,649 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:00,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,652 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,657 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:00,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:00,658 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:00,658 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:00,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:00,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405700714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405700716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,758 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:00,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:00,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:00,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:00,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:00,762 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:00,762 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,762 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:00,762 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:00,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,764 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,764 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:00,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:00,767 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:00,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:00,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:00,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:00,768 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,768 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b87bfc2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:00,771 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:00,771 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,772 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:00,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:00,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:00,779 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:00,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:00,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:00,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:00,779 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:00,780 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,780 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,781 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,782 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,786 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,785 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:00,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:00,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:00,786 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:00,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,786 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:00,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,887 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:00,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:00,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:00,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:00,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:00,891 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:00,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:00,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:00,892 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,892 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,894 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,894 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,894 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,894 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:00,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:00,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:00,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:00,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:00,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:00,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:00,904 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,904 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:00,904 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:00,904 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:00,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c8ee49[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:00,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:00,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,914 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:00,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:00,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:00,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405700919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:00,923 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:00,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:00,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:00,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:00,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:00,923 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:00,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405700919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,926 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,926 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:00,926 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:00,928 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,930 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:00,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:00,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:00,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:00,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:00,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:00,939 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:00,940 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:00,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:00,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:00,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:00,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:00,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:00,968 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/59ebf8224895460196016862a0b02fca as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/59ebf8224895460196016862a0b02fca 2023-04-25 06:54:00,971 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/9e6744dfc5b4422086f8421d276db39b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9e6744dfc5b4422086f8421d276db39b 2023-04-25 06:54:00,982 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into 9e6744dfc5b4422086f8421d276db39b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:00,982 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:00,982 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=13, startTime=1682405640511; duration=0sec 2023-04-25 06:54:00,983 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:00,983 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-25 06:54:00,982 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into 59ebf8224895460196016862a0b02fca(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:00,983 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:00,983 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=13, startTime=1682405640512; duration=0sec 2023-04-25 06:54:00,983 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:00,984 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-25 06:54:00,984 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-25 06:54:00,984 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. because compaction request was cancelled 2023-04-25 06:54:00,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/f489843f5cb2425f8578a115e715e985 2023-04-25 06:54:01,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/18ccd4a7ae4b4012a8e8736c628648ff 2023-04-25 06:54:01,054 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:01,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:01,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:01,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:01,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:01,064 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:01,064 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,064 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:01,064 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:01,065 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,065 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,066 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,067 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,067 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:01,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:01,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:01,072 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,072 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:01,072 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:01,072 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:01,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:01,074 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:01,077 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:01,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa0fa40[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:01,079 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:01,079 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,080 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:01,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:01,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:01,085 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:01,088 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:01,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:01,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:01,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:01,088 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:01,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:01,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:01,098 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:01,098 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:01,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:01,198 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:01,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:01,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:01,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:01,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:01,202 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:01,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:01,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:01,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,204 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:01,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:01,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:01,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:01,206 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,206 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:01,206 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:01,206 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:01,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:01,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:01,210 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,210 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:01,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4e1bd5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:01,213 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:01,213 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,214 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,214 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:01,214 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:01,214 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:01,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:01,215 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:01,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:01,218 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:01,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:01,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:01,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:01,218 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:01,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,221 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,222 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:01,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:01,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,227 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:01,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405701227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405701227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:01,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:01,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:01,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:01,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:01,330 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:01,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:01,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:01,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,332 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,332 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,332 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:01,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:01,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:01,333 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:01,333 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,333 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:01,334 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:01,334 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:01,334 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,334 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:01,334 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:01,334 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,334 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:01,334 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:01,334 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,334 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:01,334 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:01,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,337 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,337 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,337 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:01,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:01,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6991c451[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:01,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:01,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,339 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:01,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:01,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:01,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:01,345 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:01,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:01,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:01,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:01,345 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:01,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,348 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:01,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:01,356 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,356 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,356 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:01,356 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:01,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43096 deadline: 1682405701438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43090 deadline: 1682405701438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,442 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:54:01,444 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:54:01,456 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:01,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:01,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:01,460 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:01,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:01,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:01,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,461 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,461 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:01,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:01,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:01,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,464 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:01,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43068 deadline: 1682405701463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,464 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,464 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:01,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/a6bab4ef600e4363a403245d9d6cb12c 2023-04-25 06:54:01,464 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:01,464 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:01,464 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,465 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:01,465 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:01,465 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:01,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:01,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:01,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:01,467 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8218 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:54:01,468 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19dbc58c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:01,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:01,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/f489843f5cb2425f8578a115e715e985 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f489843f5cb2425f8578a115e715e985 2023-04-25 06:54:01,474 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:01,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:01,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:01,476 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:01,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:01,477 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:01,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:01,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:01,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:01,477 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:01,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:01,480 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:01,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:01,487 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:01,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:01,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:01,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,492 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:01,492 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:01,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:01,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:01,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f489843f5cb2425f8578a115e715e985, entries=200, sequenceid=311, filesize=14.3 K 2023-04-25 06:54:01,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/18ccd4a7ae4b4012a8e8736c628648ff as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/18ccd4a7ae4b4012a8e8736c628648ff 2023-04-25 06:54:01,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/18ccd4a7ae4b4012a8e8736c628648ff, entries=150, sequenceid=311, filesize=11.9 K 2023-04-25 06:54:01,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/a6bab4ef600e4363a403245d9d6cb12c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a6bab4ef600e4363a403245d9d6cb12c 2023-04-25 06:54:01,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a6bab4ef600e4363a403245d9d6cb12c, entries=150, sequenceid=311, filesize=11.9 K 2023-04-25 06:54:01,524 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 266734ad02022a1c1fca6ab96809adb8 in 948ms, sequenceid=311, compaction requested=true 2023-04-25 06:54:01,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:01,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:01,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:01,524 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-25 06:54:01,524 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-25 06:54:01,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:01,526 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-25 06:54:01,526 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-25 06:54:01,526 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. because compaction request was cancelled 2023-04-25 06:54:01,526 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:01,526 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-25 06:54:01,527 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-25 06:54:01,527 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. because compaction request was cancelled 2023-04-25 06:54:01,527 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:01,528 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:54:01,528 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,528 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d7687ccd70f04ab4823c4e22dd583e4d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/b50aa104e77c49dba8ac93e30fc4a978, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a6bab4ef600e4363a403245d9d6cb12c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=36.3 K 2023-04-25 06:54:01,528 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d7687ccd70f04ab4823c4e22dd583e4d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682405639371 2023-04-25 06:54:01,529 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b50aa104e77c49dba8ac93e30fc4a978, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682405639795 2023-04-25 06:54:01,529 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a6bab4ef600e4363a403245d9d6cb12c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682405639930 2023-04-25 06:54:01,539 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#68 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:01,592 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:01,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:01,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:01,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:01,596 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:01,596 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,596 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:01,596 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:01,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:01,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:01,598 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:01,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:01,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:01,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:01,601 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:01,601 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:01,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:01,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:01,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405701749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405701751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405701852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:01,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405701859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:01,958 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/9aa1796ad7a24e7a9345be7b6e5b7b1e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9aa1796ad7a24e7a9345be7b6e5b7b1e 2023-04-25 06:54:01,967 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into 9aa1796ad7a24e7a9345be7b6e5b7b1e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:01,967 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:01,967 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=13, startTime=1682405641524; duration=0sec 2023-04-25 06:54:01,967 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:02,015 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/76e2764d821c43e68b1e406bb8a2afd1 2023-04-25 06:54:02,027 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5bfada28234e45e8a1fa022748c17ef5 2023-04-25 06:54:02,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:02,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405702054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:02,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405702061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:02,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405702356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:02,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405702366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,443 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/635082143bc34fdfb6143c4b505fb0e9 2023-04-25 06:54:02,453 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/76e2764d821c43e68b1e406bb8a2afd1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/76e2764d821c43e68b1e406bb8a2afd1 2023-04-25 06:54:02,466 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/76e2764d821c43e68b1e406bb8a2afd1, entries=150, sequenceid=333, filesize=11.9 K 2023-04-25 06:54:02,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/5bfada28234e45e8a1fa022748c17ef5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bfada28234e45e8a1fa022748c17ef5 2023-04-25 06:54:02,480 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bfada28234e45e8a1fa022748c17ef5, entries=150, sequenceid=333, filesize=11.9 K 2023-04-25 06:54:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/635082143bc34fdfb6143c4b505fb0e9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/635082143bc34fdfb6143c4b505fb0e9 2023-04-25 06:54:02,492 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/635082143bc34fdfb6143c4b505fb0e9, entries=150, sequenceid=333, filesize=11.9 K 2023-04-25 06:54:02,493 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 266734ad02022a1c1fca6ab96809adb8 in 892ms, sequenceid=333, compaction requested=true 2023-04-25 06:54:02,494 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:02,494 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:02,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:02,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:02,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:02,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:02,496 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:02,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:02,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b14e0d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:02,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:02,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:02,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,503 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:02,503 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:02,508 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:02,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:02,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:02,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:02,508 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:02,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:02,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:02,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,533 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,536 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,536 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:02,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:02,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:02,536 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:02,536 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:02,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:02,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:02,536 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:02,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:02,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:02,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:02,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:02,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:02,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:02,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:02,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:02,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:02,660 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:02,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:02,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:02,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:02,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:02,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:02,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:02,663 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:02,663 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,663 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:02,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:02,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:02,688 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:02,688 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:02,688 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:54:02,688 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:02,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:02,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:02,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:02,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:02,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:02,735 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/0674064b435d4371ad8945358edfae62 2023-04-25 06:54:02,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:02,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:02,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:02,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405702915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:02,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405702919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405703021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405703026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,150 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/22eec07ede3b43939d42131e85205704 2023-04-25 06:54:03,167 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/509ad1e5ffc8437b8cea1045fdcb6edf 2023-04-25 06:54:03,174 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/0674064b435d4371ad8945358edfae62 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/0674064b435d4371ad8945358edfae62 2023-04-25 06:54:03,185 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/0674064b435d4371ad8945358edfae62, entries=150, sequenceid=346, filesize=11.9 K 2023-04-25 06:54:03,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/22eec07ede3b43939d42131e85205704 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/22eec07ede3b43939d42131e85205704 2023-04-25 06:54:03,200 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/22eec07ede3b43939d42131e85205704, entries=150, sequenceid=346, filesize=11.9 K 2023-04-25 06:54:03,203 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/509ad1e5ffc8437b8cea1045fdcb6edf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/509ad1e5ffc8437b8cea1045fdcb6edf 2023-04-25 06:54:03,220 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/509ad1e5ffc8437b8cea1045fdcb6edf, entries=150, sequenceid=346, filesize=11.9 K 2023-04-25 06:54:03,222 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 266734ad02022a1c1fca6ab96809adb8 in 533ms, sequenceid=346, compaction requested=true 2023-04-25 06:54:03,222 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:03,222 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:03,224 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,225 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,226 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,226 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405642903, max timestamp=1682405643226Num uniques 859; 2023-04-25 06:54:03,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:03,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:03,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:03,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e1e6d5d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:03,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405642903, max timestamp=1682405643231Num uniques 859; 2023-04-25 06:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:03,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:03,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=64.85 KB heapSize=184.09 KB 2023-04-25 06:54:03,234 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:03,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:03,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:03,238 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:03,256 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:03,256 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:03,256 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:03,257 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:03,257 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:03,259 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,259 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,259 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,260 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:03,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:03,280 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:03,280 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:03,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405703309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405703317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,380 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:03,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:03,385 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:03,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:03,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:03,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,390 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:03,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:03,392 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,392 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:03,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:03,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:03,394 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,394 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:03,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ca9583[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:03,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:03,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,397 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:03,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:03,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:03,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:03,400 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:03,401 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,401 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:03,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:03,404 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:03,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:03,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:03,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:03,404 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:03,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:03,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:03,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,411 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,411 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:03,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405703418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405703425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,511 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:03,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:03,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:03,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:03,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:03,515 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:03,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:03,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:03,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,518 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:03,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:03,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:03,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:03,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:03,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:03,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:03,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:03,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:03,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:03,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:03,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:03,523 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,524 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,524 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1efacc93[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:03,525 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:03,525 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:03,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:03,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:03,526 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:03,528 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:03,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:03,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:03,530 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:03,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:03,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:03,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:03,530 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:03,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,533 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,534 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:03,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:03,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:03,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,538 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,538 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,538 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:03,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405703624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405703630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,639 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:03,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:03,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:03,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:03,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:03,644 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:03,644 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,644 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:03,644 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:03,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:03,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:03,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:03,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:03,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:03,664 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:03,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b03867[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:03,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:03,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:03,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:03,674 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:03,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:03,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:03,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:03,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:03,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:03,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:03,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:03,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:03,690 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:03,691 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:03,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:03,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/ea85a7343b3f40bfbff6ce461a33f364 2023-04-25 06:54:03,791 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:03,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:03,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:03,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:03,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:03,795 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:03,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:03,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:03,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,797 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:03,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:03,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:03,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:03,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:03,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:03,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:03,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:03,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:03,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:03,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:03,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:03,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:03,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,801 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:03,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5809cb09[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:03,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:03,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,877 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:03,877 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:03,877 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:03,877 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:03,878 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,878 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:03,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:03,922 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:03,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:03,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:03,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:03,922 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:03,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405703929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:03,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405703933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:03,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:03,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:03,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,019 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,019 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:04,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:04,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:04,019 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,019 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,019 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,019 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:04,019 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:04,020 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,020 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,020 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,020 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,120 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:04,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:04,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:04,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/e5f26c2581a24533806ed055e8421143 2023-04-25 06:54:04,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:04,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:04,124 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:04,124 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,124 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:04,124 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:04,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/3a89a66db45448048c0eb862ba56f81c 2023-04-25 06:54:04,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/ea85a7343b3f40bfbff6ce461a33f364 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/ea85a7343b3f40bfbff6ce461a33f364 2023-04-25 06:54:04,150 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,150 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,150 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:04,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/ea85a7343b3f40bfbff6ce461a33f364, entries=250, sequenceid=368, filesize=16.7 K 2023-04-25 06:54:04,152 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/e5f26c2581a24533806ed055e8421143 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e5f26c2581a24533806ed055e8421143 2023-04-25 06:54:04,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:04,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:04,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:04,154 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:04,154 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,154 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:04,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e5f26c2581a24533806ed055e8421143, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 06:54:04,161 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,161 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:04,161 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:04,161 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:04,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/3a89a66db45448048c0eb862ba56f81c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3a89a66db45448048c0eb862ba56f81c 2023-04-25 06:54:04,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3a89a66db45448048c0eb862ba56f81c, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 06:54:04,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~64.85 KB/66410, heapSize ~184.05 KB/188464, currentSize=114.05 KB/116790 for 266734ad02022a1c1fca6ab96809adb8 in 934ms, sequenceid=368, compaction requested=true 2023-04-25 06:54:04,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:04,169 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:04,169 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:04,172 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 68966 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:04,172 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:54:04,172 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,172 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9e6744dfc5b4422086f8421d276db39b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f489843f5cb2425f8578a115e715e985, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/76e2764d821c43e68b1e406bb8a2afd1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/0674064b435d4371ad8945358edfae62, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/ea85a7343b3f40bfbff6ce461a33f364] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=67.3 K 2023-04-25 06:54:04,173 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:04,173 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:54:04,173 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9e6744dfc5b4422086f8421d276db39b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682405639795 2023-04-25 06:54:04,173 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,173 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/59ebf8224895460196016862a0b02fca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/18ccd4a7ae4b4012a8e8736c628648ff, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bfada28234e45e8a1fa022748c17ef5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/22eec07ede3b43939d42131e85205704, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e5f26c2581a24533806ed055e8421143] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=60.2 K 2023-04-25 06:54:04,173 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting f489843f5cb2425f8578a115e715e985, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682405639930 2023-04-25 06:54:04,173 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 59ebf8224895460196016862a0b02fca, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682405639795 2023-04-25 06:54:04,174 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 76e2764d821c43e68b1e406bb8a2afd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682405640594 2023-04-25 06:54:04,174 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 18ccd4a7ae4b4012a8e8736c628648ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682405639930 2023-04-25 06:54:04,174 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 0674064b435d4371ad8945358edfae62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1682405641741 2023-04-25 06:54:04,174 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5bfada28234e45e8a1fa022748c17ef5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682405640594 2023-04-25 06:54:04,175 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting ea85a7343b3f40bfbff6ce461a33f364, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682405642903 2023-04-25 06:54:04,175 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 22eec07ede3b43939d42131e85205704, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1682405641741 2023-04-25 06:54:04,175 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e5f26c2581a24533806ed055e8421143, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682405642917 2023-04-25 06:54:04,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:04,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55623b06[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:04,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:04,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,191 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:04,191 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#78 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:04,216 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/71ae4791bbb7489bbb7a3d6d54cae6ef as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/71ae4791bbb7489bbb7a3d6d54cae6ef 2023-04-25 06:54:04,223 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into 71ae4791bbb7489bbb7a3d6d54cae6ef(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:04,223 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:04,223 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=11, startTime=1682405644169; duration=0sec 2023-04-25 06:54:04,223 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:04,223 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:04,225 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:04,225 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:54:04,225 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,225 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9aa1796ad7a24e7a9345be7b6e5b7b1e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/635082143bc34fdfb6143c4b505fb0e9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/509ad1e5ffc8437b8cea1045fdcb6edf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3a89a66db45448048c0eb862ba56f81c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=48.3 K 2023-04-25 06:54:04,225 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9aa1796ad7a24e7a9345be7b6e5b7b1e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682405639930 2023-04-25 06:54:04,226 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 635082143bc34fdfb6143c4b505fb0e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682405640594 2023-04-25 06:54:04,226 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 509ad1e5ffc8437b8cea1045fdcb6edf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1682405641741 2023-04-25 06:54:04,227 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 3a89a66db45448048c0eb862ba56f81c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682405642917 2023-04-25 06:54:04,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:04,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:04,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:04,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:04,236 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#80 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:04,247 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/a261d2fa3d2f4c009e79283b3d294ed4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a261d2fa3d2f4c009e79283b3d294ed4 2023-04-25 06:54:04,254 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into a261d2fa3d2f4c009e79283b3d294ed4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:04,254 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:04,254 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=12, startTime=1682405644169; duration=0sec 2023-04-25 06:54:04,254 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:04,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:04,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:04,276 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:04,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:04,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:04,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:04,276 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:04,311 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,313 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:04,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:04,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:04,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,343 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,343 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,343 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,343 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,343 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:04,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,343 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:04,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-25 06:54:04,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:04,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:04,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:04,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:04,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:04,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:04,444 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:04,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:04,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:04,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:04,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405704454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:04,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:04,460 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:04,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:04,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:04,461 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,461 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:04,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405704457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/3ab6f00bc62d49128bdd5f287159ca51 2023-04-25 06:54:04,498 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,498 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:04,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:04,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:04,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:04,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:04,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:04,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:04,510 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,510 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:04,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:04,510 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:04,510 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:04,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:04,510 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,511 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:04,511 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:04,512 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,512 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,513 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,513 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:04,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1db4eb6d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:04,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:04,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,527 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:04,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:04,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:04,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:04,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:04,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:04,537 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:04,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:04,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:04,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:04,537 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:04,550 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,553 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:04,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,558 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:04,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:04,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:04,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405704558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:04,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405704563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,605 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/3636701b058b48a58cf22bc2c7a4fb4d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/3636701b058b48a58cf22bc2c7a4fb4d 2023-04-25 06:54:04,611 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into 3636701b058b48a58cf22bc2c7a4fb4d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:04,611 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:04,611 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=11, startTime=1682405644169; duration=0sec 2023-04-25 06:54:04,612 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:04,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:04,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:04,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:04,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:04,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:04,662 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:04,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:04,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:04,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,663 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:04,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:04,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:04,665 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,665 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:04,666 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:04,666 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:04,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:04,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:04,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:04,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,667 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:04,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:04,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9538bbc[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:04,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,673 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:04,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:04,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:04,673 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,673 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:04,676 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,676 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:04,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:04,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:04,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:04,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:04,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:04,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:04,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:04,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:04,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,687 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:04,687 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:04,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:04,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405704763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:04,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405704766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,787 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:04,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:04,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:04,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:04,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:04,791 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:04,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:04,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:04,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:04,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:04,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:04,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:04,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:04,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:04,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:04,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:04,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:04,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:04,797 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:04,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:04,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f94581c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:04,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:04,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:04,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:04,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:04,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:04,812 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:04,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:04,815 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:04,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:04,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:04,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:04,815 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:04,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:04,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:04,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:04,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:04,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:04,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:04,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:04,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:04,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:04,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:04,825 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:04,825 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:04,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:04,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/a6c3e6762e5741359b3268f1f3995146 2023-04-25 06:54:04,948 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:04,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:04,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:04,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:04,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:04,972 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:04,972 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,972 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:04,972 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:04,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,974 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:04,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:04,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:04,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:04,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:04,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:04,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:04,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:04,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:04,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:05,008 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:05,009 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,009 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:05,009 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:05,010 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:05,010 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,010 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:05,010 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:05,010 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,010 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:05,010 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:05,016 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:05,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:05,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5beafae3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:05,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:05,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,023 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:05,025 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:05,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:05,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:05,026 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:05,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:05,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:05,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:05,026 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:05,029 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,029 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,029 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:05,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:05,037 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,037 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,037 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,037 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,036 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,037 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:05,037 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:05,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405705067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405705068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,137 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:05,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:05,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:05,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:05,144 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:05,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:05,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:05,145 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,145 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,146 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:05,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:05,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:05,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:05,156 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,157 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:05,157 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:05,157 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:05,159 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,159 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,159 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,159 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:05,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@613f9d70[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:05,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:05,162 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,163 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:05,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:05,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:05,164 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,164 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:05,171 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:05,171 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,171 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:05,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:05,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:05,176 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:05,176 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:05,176 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:05,176 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:05,176 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:05,183 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,183 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,183 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:05,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:05,193 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:05,193 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,192 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:05,293 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:05,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:05,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:05,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:05,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:05,296 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:05,296 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,296 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:05,296 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:05,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,299 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,299 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,299 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:05,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:05,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:05,301 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,301 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:05,301 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:05,301 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,302 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:05,302 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:05,303 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:05,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74a4f480[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:05,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:05,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:05,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:05,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:05,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:05,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:05,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:05,320 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:05,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:05,330 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:05,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:05,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:05,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:05,330 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:05,340 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,340 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,340 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,344 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,348 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:05,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:05,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:05,355 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:05,355 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:05,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/229f5f07fd744826a4123be0efc3524c 2023-04-25 06:54:05,419 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/3ab6f00bc62d49128bdd5f287159ca51 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3ab6f00bc62d49128bdd5f287159ca51 2023-04-25 06:54:05,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3ab6f00bc62d49128bdd5f287159ca51, entries=200, sequenceid=391, filesize=14.3 K 2023-04-25 06:54:05,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/a6c3e6762e5741359b3268f1f3995146 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6c3e6762e5741359b3268f1f3995146 2023-04-25 06:54:05,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6c3e6762e5741359b3268f1f3995146, entries=150, sequenceid=391, filesize=11.9 K 2023-04-25 06:54:05,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/229f5f07fd744826a4123be0efc3524c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/229f5f07fd744826a4123be0efc3524c 2023-04-25 06:54:05,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/229f5f07fd744826a4123be0efc3524c, entries=150, sequenceid=391, filesize=11.9 K 2023-04-25 06:54:05,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 266734ad02022a1c1fca6ab96809adb8 in 1016ms, sequenceid=391, compaction requested=false 2023-04-25 06:54:05,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:05,455 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:05,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:05,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:05,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:05,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:05,461 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:05,461 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:05,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:05,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,464 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:05,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:05,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:05,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:05,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:05,472 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,472 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:05,472 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 06:54:05,472 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:05,473 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:05,473 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:05,473 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:05,473 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:05,473 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:05,496 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/2d0ab03f6ad64484a3ee06e3a4a48fd3 2023-04-25 06:54:05,532 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/13bd2139056e40c6b155f904781ec2e5 2023-04-25 06:54:05,546 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/32d57850a002411381ce40b87015a40f 2023-04-25 06:54:05,553 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/2d0ab03f6ad64484a3ee06e3a4a48fd3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2d0ab03f6ad64484a3ee06e3a4a48fd3 2023-04-25 06:54:05,561 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2d0ab03f6ad64484a3ee06e3a4a48fd3, entries=150, sequenceid=403, filesize=11.9 K 2023-04-25 06:54:05,562 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/13bd2139056e40c6b155f904781ec2e5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/13bd2139056e40c6b155f904781ec2e5 2023-04-25 06:54:05,568 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/13bd2139056e40c6b155f904781ec2e5, entries=150, sequenceid=403, filesize=11.9 K 2023-04-25 06:54:05,569 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/32d57850a002411381ce40b87015a40f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/32d57850a002411381ce40b87015a40f 2023-04-25 06:54:05,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:05,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:05,575 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/32d57850a002411381ce40b87015a40f, entries=150, sequenceid=403, filesize=11.9 K 2023-04-25 06:54:05,576 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=6.71 KB/6870 for 266734ad02022a1c1fca6ab96809adb8 in 104ms, sequenceid=403, compaction requested=true 2023-04-25 06:54:05,576 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:05,576 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:05,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:05,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:05,577 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:05,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cd06b0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:05,582 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:05,582 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,584 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:05,584 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:05,584 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:05,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:05,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:05,586 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:05,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:54:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:05,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:05,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:05,589 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:05,589 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:05,589 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:05,589 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:05,589 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:05,591 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,591 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,591 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,595 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,598 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:05,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:05,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,599 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,599 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,599 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,599 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:05,599 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:05,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405645607, max timestamp=1682405645618Num uniques 859; 2023-04-25 06:54:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405645607, max timestamp=1682405645618Num uniques 859; 2023-04-25 06:54:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:05,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405705649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405705649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,699 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:05,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:05,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:05,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:05,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:05,703 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:05,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:05,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:05,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,706 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,706 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,707 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,707 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:05,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:05,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:05,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:05,708 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,708 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:05,709 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:05,709 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:05,711 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,711 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,711 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:05,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45fcc088[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:05,713 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:05,713 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,715 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:05,715 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:05,715 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:05,715 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,715 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:05,718 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:05,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:05,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:05,722 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:05,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:05,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:05,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:05,722 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:05,725 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,725 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,725 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,725 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,727 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,733 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:05,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:05,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,734 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,734 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,734 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:05,734 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:05,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405705755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405705755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,834 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:05,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:05,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:05,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:05,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:05,838 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:05,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:05,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:05,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,843 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,843 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,843 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,843 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:05,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:05,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:05,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:05,845 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,845 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:05,845 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:05,845 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,852 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:05,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b5115a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:05,854 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:05,855 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,858 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,858 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:05,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:05,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:05,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:05,862 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:05,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,862 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,862 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:05,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:05,867 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:05,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:05,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:05,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:05,867 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:05,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,871 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,880 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,882 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:05,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,885 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:05,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:05,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,886 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:05,886 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:05,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:05,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405705956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405705958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,986 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:05,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:05,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:05,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:05,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:05,990 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:05,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:05,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:05,991 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,991 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,993 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:05,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:05,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:05,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:05,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:05,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:05,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:05,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:05,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:05,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:05,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:05,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:05,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:05,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:05,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:05,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:05,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:05,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b66ee38[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:05,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:05,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,001 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:06,001 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:06,001 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:06,001 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,001 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:06,003 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/246aeaaacd17443fb5c9be10805716b3 2023-04-25 06:54:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:06,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:06,007 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:06,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:06,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:06,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:06,007 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:06,008 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,008 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,009 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,009 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:06,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:06,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,016 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:06,016 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:06,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/6675175fb9f544adb281fdb3821e9f1e 2023-04-25 06:54:06,116 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:06,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:06,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:06,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:06,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:06,120 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:06,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:06,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:06,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,127 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,128 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,128 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:06,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:06,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:06,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:06,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:06,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:06,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:06,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:06,138 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:06,138 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:06,138 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:06,138 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:06,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:06,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:06,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:06,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:06,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a3f8162[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:06,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:06,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,144 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,145 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,145 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:06,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:06,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:06,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:06,146 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:06,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:06,150 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:06,150 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:06,150 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:06,150 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:06,150 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:06,153 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,153 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,153 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,154 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:06,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:06,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:06,158 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:06,158 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:06,159 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,159 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,258 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:06,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:06,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:06,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:06,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:06,262 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:06,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:06,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:06,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,263 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:06,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405706261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:06,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405706261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,264 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:06,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:06,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:06,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:06,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:06,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:06,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:06,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:06,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,269 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:06,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e4fc2d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:06,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:06,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,272 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:06,272 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,272 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:06,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:06,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:06,276 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:06,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:06,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:06,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:06,276 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:06,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,279 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,285 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:06,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:06,284 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,285 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:06,285 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:06,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:06,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:06,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:06,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:06,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:06,389 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:06,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:06,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:06,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,390 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:06,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:06,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:06,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:06,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:06,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:06,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:06,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:06,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:06,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:06,394 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:06,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:06,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@792b038a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:06,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,398 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:06,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:06,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:06,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:06,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:06,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:06,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:06,404 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:06,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:06,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:06,405 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:06,405 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:06,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:06,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:06,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:06,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:06,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:06,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:06,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:06,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:06,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:06,411 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:06,411 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:06,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/c35c50a47d87409f99952ba2ac83a97f 2023-04-25 06:54:06,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/246aeaaacd17443fb5c9be10805716b3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/246aeaaacd17443fb5c9be10805716b3 2023-04-25 06:54:06,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/246aeaaacd17443fb5c9be10805716b3, entries=200, sequenceid=413, filesize=14.3 K 2023-04-25 06:54:06,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/6675175fb9f544adb281fdb3821e9f1e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/6675175fb9f544adb281fdb3821e9f1e 2023-04-25 06:54:06,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/6675175fb9f544adb281fdb3821e9f1e, entries=150, sequenceid=413, filesize=11.9 K 2023-04-25 06:54:06,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/c35c50a47d87409f99952ba2ac83a97f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/c35c50a47d87409f99952ba2ac83a97f 2023-04-25 06:54:06,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/c35c50a47d87409f99952ba2ac83a97f, entries=150, sequenceid=413, filesize=11.9 K 2023-04-25 06:54:06,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 266734ad02022a1c1fca6ab96809adb8 in 890ms, sequenceid=413, compaction requested=true 2023-04-25 06:54:06,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:06,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:06,478 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:06,478 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:06,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:06,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:06,480 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54510 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:06,481 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:54:06,481 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,481 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:06,481 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/71ae4791bbb7489bbb7a3d6d54cae6ef, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3ab6f00bc62d49128bdd5f287159ca51, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2d0ab03f6ad64484a3ee06e3a4a48fd3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/246aeaaacd17443fb5c9be10805716b3] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=53.2 K 2023-04-25 06:54:06,481 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:54:06,481 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,481 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/3636701b058b48a58cf22bc2c7a4fb4d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6c3e6762e5741359b3268f1f3995146, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/13bd2139056e40c6b155f904781ec2e5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/6675175fb9f544adb281fdb3821e9f1e] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=48.5 K 2023-04-25 06:54:06,482 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 71ae4791bbb7489bbb7a3d6d54cae6ef, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682405642917 2023-04-25 06:54:06,482 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3636701b058b48a58cf22bc2c7a4fb4d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682405642917 2023-04-25 06:54:06,482 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 3ab6f00bc62d49128bdd5f287159ca51, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1682405643305 2023-04-25 06:54:06,482 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a6c3e6762e5741359b3268f1f3995146, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1682405643305 2023-04-25 06:54:06,483 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 2d0ab03f6ad64484a3ee06e3a4a48fd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682405644441 2023-04-25 06:54:06,483 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 13bd2139056e40c6b155f904781ec2e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682405644441 2023-04-25 06:54:06,485 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6675175fb9f544adb281fdb3821e9f1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405645581 2023-04-25 06:54:06,485 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 246aeaaacd17443fb5c9be10805716b3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405645581 2023-04-25 06:54:06,494 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:06,498 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:06,511 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:06,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:06,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:06,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:06,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:06,514 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:06,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:06,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:06,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:06,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:06,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:06,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:06,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:06,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:06,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:06,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:06,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:06,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:06,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:06,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:06,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:06,517 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:06,518 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=136.42 KB heapSize=393.47 KB 2023-04-25 06:54:06,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:06,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:06,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:06,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:06,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:06,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:06,533 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/4cf1ce40b90a4f2e89d2d3252a4610d1 2023-04-25 06:54:06,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:06,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:06,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:06,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405706778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:06,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405706780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:06,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405706882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:06,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405706883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,910 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/03c728301a22450c950943ef32bf838f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/03c728301a22450c950943ef32bf838f 2023-04-25 06:54:06,910 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/9ed770c8d1654394abf33cbb6286c1e4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9ed770c8d1654394abf33cbb6286c1e4 2023-04-25 06:54:06,916 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into 03c728301a22450c950943ef32bf838f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:06,916 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into 9ed770c8d1654394abf33cbb6286c1e4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:06,916 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:06,916 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:06,916 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=12, startTime=1682405646478; duration=0sec 2023-04-25 06:54:06,916 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=12, startTime=1682405646478; duration=0sec 2023-04-25 06:54:06,916 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:06,917 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:06,917 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:06,918 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:06,918 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:54:06,919 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,919 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a261d2fa3d2f4c009e79283b3d294ed4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/229f5f07fd744826a4123be0efc3524c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/32d57850a002411381ce40b87015a40f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/c35c50a47d87409f99952ba2ac83a97f] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=48.5 K 2023-04-25 06:54:06,919 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a261d2fa3d2f4c009e79283b3d294ed4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682405642917 2023-04-25 06:54:06,920 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 229f5f07fd744826a4123be0efc3524c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1682405643305 2023-04-25 06:54:06,921 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 32d57850a002411381ce40b87015a40f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682405644441 2023-04-25 06:54:06,921 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c35c50a47d87409f99952ba2ac83a97f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405645581 2023-04-25 06:54:06,929 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#94 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:06,955 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/948635f0373f4be7ad14d4abce0696d2 2023-04-25 06:54:06,976 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/650b90356f3742ef858ded0d25d4b3a6 2023-04-25 06:54:06,981 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/4cf1ce40b90a4f2e89d2d3252a4610d1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4cf1ce40b90a4f2e89d2d3252a4610d1 2023-04-25 06:54:06,986 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4cf1ce40b90a4f2e89d2d3252a4610d1, entries=150, sequenceid=445, filesize=11.9 K 2023-04-25 06:54:06,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/948635f0373f4be7ad14d4abce0696d2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/948635f0373f4be7ad14d4abce0696d2 2023-04-25 06:54:06,991 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/948635f0373f4be7ad14d4abce0696d2, entries=150, sequenceid=445, filesize=11.9 K 2023-04-25 06:54:06,992 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/650b90356f3742ef858ded0d25d4b3a6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/650b90356f3742ef858ded0d25d4b3a6 2023-04-25 06:54:06,998 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/650b90356f3742ef858ded0d25d4b3a6, entries=150, sequenceid=445, filesize=11.9 K 2023-04-25 06:54:06,999 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~393.42 KB/402864, currentSize=40.25 KB/41220 for 266734ad02022a1c1fca6ab96809adb8 in 481ms, sequenceid=445, compaction requested=false 2023-04-25 06:54:06,999 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:06,999 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:06,999 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:07,002 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:07,005 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@458025b9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:07,005 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:07,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:07,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:07,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:07,008 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:07,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:07,010 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:07,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:07,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:07,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:07,011 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:07,012 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,012 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,012 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,013 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,017 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,018 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:07,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:07,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:07,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:07,018 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:07,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:07,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:54:07,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:07,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:07,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:07,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:07,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:07,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:07,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:07,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:07,122 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:07,122 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:07,122 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:07,122 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:07,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:07,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:07,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:07,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:07,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:07,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:07,133 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405647108, max timestamp=1682405647125Num uniques 859; 2023-04-25 06:54:07,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:07,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405647108, max timestamp=1682405647125Num uniques 859; 2023-04-25 06:54:07,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405647108, max timestamp=1682405647127Num uniques 859; 2023-04-25 06:54:07,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,134 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,134 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:07,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:07,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@136288fa[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:07,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:07,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,137 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:07,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:07,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:07,139 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:07,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:07,142 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:07,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:07,142 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:07,145 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,145 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,145 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:07,150 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:07,150 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:07,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:07,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:07,151 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:07,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,151 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:07,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405707162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405707164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,251 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:07,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:07,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:07,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:07,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:07,255 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:07,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:07,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:07,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:07,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,256 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,256 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:07,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:07,257 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,257 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:07,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:07,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:07,258 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:07,258 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:07,259 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:07,259 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:07,260 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:07,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:07,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d490b2f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:07,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,264 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,264 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:07,264 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:07,264 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:07,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:07,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:07,265 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:07,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:07,268 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:07,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:07,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:07,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:07,268 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:07,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405707268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,270 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,270 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,270 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405707270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,272 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:07,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:07,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:07,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:07,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:07,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:07,275 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:07,275 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:07,342 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/934a1745d4904c4b93efb3b2c2efe847 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/934a1745d4904c4b93efb3b2c2efe847 2023-04-25 06:54:07,349 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into 934a1745d4904c4b93efb3b2c2efe847(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:07,349 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:07,349 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=12, startTime=1682405646478; duration=0sec 2023-04-25 06:54:07,350 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:07,375 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:07,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:07,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:07,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:07,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:07,379 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:07,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:07,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:07,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:07,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,382 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:07,382 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:07,382 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75f5a5e6 to 127.0.0.1:60416 2023-04-25 06:54:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:07,383 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:07,383 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,383 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:07,384 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ae6c42b to 127.0.0.1:60416 2023-04-25 06:54:07,384 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,385 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71f9f426 to 127.0.0.1:60416 2023-04-25 06:54:07,385 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:07,386 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x726fb449 to 127.0.0.1:60416 2023-04-25 06:54:07,386 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:07,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:07,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:07,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:07,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:07,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:07,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:07,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:07,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:07,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:07,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:07,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. started... 2023-04-25 06:54:07,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. as already flushing 2023-04-25 06:54:07,393 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:07,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:07,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:07,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:07,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:07,394 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,394 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:07,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cdf785f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:07,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:07,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,398 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:07,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:07,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:07,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:07,399 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:07,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:07,402 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:07,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:07,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:07,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:07,402 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:07,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:07,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:07,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:07,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:07,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:07,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:07,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:07,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:07,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:07,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:07,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:07,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:07,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:07,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:07,411 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:07,411 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:07,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:07,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405707472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405707474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,511 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:07,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:07,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/26e76b381d0a4a5f9f1d525e24042d23 2023-04-25 06:54:07,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405707775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:07,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405707776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:07,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/7e6d9af2ed7f4225a0a7f7f3810898ff 2023-04-25 06:54:08,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:08,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43142 deadline: 1682405708278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:08,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:08,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43074 deadline: 1682405708282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:08,344 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/1479d520ff1f402499946568f0f96f66 2023-04-25 06:54:08,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/26e76b381d0a4a5f9f1d525e24042d23 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/26e76b381d0a4a5f9f1d525e24042d23 2023-04-25 06:54:08,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/26e76b381d0a4a5f9f1d525e24042d23, entries=150, sequenceid=457, filesize=11.9 K 2023-04-25 06:54:08,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/7e6d9af2ed7f4225a0a7f7f3810898ff as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/7e6d9af2ed7f4225a0a7f7f3810898ff 2023-04-25 06:54:08,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/7e6d9af2ed7f4225a0a7f7f3810898ff, entries=100, sequenceid=457, filesize=9.5 K 2023-04-25 06:54:08,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/1479d520ff1f402499946568f0f96f66 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1479d520ff1f402499946568f0f96f66 2023-04-25 06:54:08,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1479d520ff1f402499946568f0f96f66, entries=100, sequenceid=457, filesize=9.5 K 2023-04-25 06:54:08,366 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 266734ad02022a1c1fca6ab96809adb8 in 1279ms, sequenceid=457, compaction requested=true 2023-04-25 06:54:08,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:08,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:08,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:08,366 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:08,366 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:08,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:08,367 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37576 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:08,367 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:08,367 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/A is initiating minor compaction (all files) 2023-04-25 06:54:08,367 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/B is initiating minor compaction (all files) 2023-04-25 06:54:08,367 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/A in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:08,367 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/B in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:08,367 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9ed770c8d1654394abf33cbb6286c1e4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4cf1ce40b90a4f2e89d2d3252a4610d1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/26e76b381d0a4a5f9f1d525e24042d23] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=36.7 K 2023-04-25 06:54:08,367 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/03c728301a22450c950943ef32bf838f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/948635f0373f4be7ad14d4abce0696d2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/7e6d9af2ed7f4225a0a7f7f3810898ff] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=34.3 K 2023-04-25 06:54:08,368 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9ed770c8d1654394abf33cbb6286c1e4, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405645581 2023-04-25 06:54:08,368 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 03c728301a22450c950943ef32bf838f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405645581 2023-04-25 06:54:08,368 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 4cf1ce40b90a4f2e89d2d3252a4610d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405645630 2023-04-25 06:54:08,368 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 948635f0373f4be7ad14d4abce0696d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405645630 2023-04-25 06:54:08,369 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 26e76b381d0a4a5f9f1d525e24042d23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682405646766 2023-04-25 06:54:08,369 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7e6d9af2ed7f4225a0a7f7f3810898ff, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682405646766 2023-04-25 06:54:08,375 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#A#compaction#99 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:08,377 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#B#compaction#100 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:08,386 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/36f7b2dda1d14f29bad7c775c688bdf3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/36f7b2dda1d14f29bad7c775c688bdf3 2023-04-25 06:54:08,391 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/B of 266734ad02022a1c1fca6ab96809adb8 into 36f7b2dda1d14f29bad7c775c688bdf3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:08,391 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:08,391 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/B, priority=13, startTime=1682405648366; duration=0sec 2023-04-25 06:54:08,391 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:08,391 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:08,392 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:08,393 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 266734ad02022a1c1fca6ab96809adb8/C is initiating minor compaction (all files) 2023-04-25 06:54:08,393 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 266734ad02022a1c1fca6ab96809adb8/C in TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:08,393 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/934a1745d4904c4b93efb3b2c2efe847, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/650b90356f3742ef858ded0d25d4b3a6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1479d520ff1f402499946568f0f96f66] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp, totalSize=34.3 K 2023-04-25 06:54:08,393 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 934a1745d4904c4b93efb3b2c2efe847, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405645581 2023-04-25 06:54:08,393 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 650b90356f3742ef858ded0d25d4b3a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405645630 2023-04-25 06:54:08,394 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1479d520ff1f402499946568f0f96f66, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682405646766 2023-04-25 06:54:08,399 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 266734ad02022a1c1fca6ab96809adb8#C#compaction#101 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:08,786 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/229cf56ec9a7488d9d0a867d555e888a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/229cf56ec9a7488d9d0a867d555e888a 2023-04-25 06:54:08,793 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/A of 266734ad02022a1c1fca6ab96809adb8 into 229cf56ec9a7488d9d0a867d555e888a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:08,793 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:08,793 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/A, priority=13, startTime=1682405648366; duration=0sec 2023-04-25 06:54:08,793 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:08,811 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/4683f3a6bc864b00b71bed06baca0c27 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4683f3a6bc864b00b71bed06baca0c27 2023-04-25 06:54:08,817 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 266734ad02022a1c1fca6ab96809adb8/C of 266734ad02022a1c1fca6ab96809adb8 into 4683f3a6bc864b00b71bed06baca0c27(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:08,817 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:08,817 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8., storeName=266734ad02022a1c1fca6ab96809adb8/C, priority=13, startTime=1682405648366; duration=0sec 2023-04-25 06:54:08,817 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:09,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:09,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-25 06:54:09,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:09,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 06:54:09,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:09,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 06:54:09,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:09,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 06:54:09,285 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ca4a062 to 127.0.0.1:60416 2023-04-25 06:54:09,285 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:09,290 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e8196e0 to 127.0.0.1:60416 2023-04-25 06:54:09,291 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:09,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/f146d982316d41baba5ea5fcb44db23f 2023-04-25 06:54:10,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/ed08a12e3b9d4d208ddaa208ab216726 2023-04-25 06:54:10,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/1d482d179762440bb34e25e76beabdd5 2023-04-25 06:54:10,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/f146d982316d41baba5ea5fcb44db23f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f146d982316d41baba5ea5fcb44db23f 2023-04-25 06:54:10,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f146d982316d41baba5ea5fcb44db23f, entries=150, sequenceid=499, filesize=11.9 K 2023-04-25 06:54:10,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/ed08a12e3b9d4d208ddaa208ab216726 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/ed08a12e3b9d4d208ddaa208ab216726 2023-04-25 06:54:10,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/ed08a12e3b9d4d208ddaa208ab216726, entries=150, sequenceid=499, filesize=11.9 K 2023-04-25 06:54:10,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/1d482d179762440bb34e25e76beabdd5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1d482d179762440bb34e25e76beabdd5 2023-04-25 06:54:10,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1d482d179762440bb34e25e76beabdd5, entries=150, sequenceid=499, filesize=11.9 K 2023-04-25 06:54:10,547 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=6.71 KB/6870 for 266734ad02022a1c1fca6ab96809adb8 in 1263ms, sequenceid=499, compaction requested=false 2023-04-25 06:54:10,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:11,447 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2411a862 to 127.0.0.1:60416 2023-04-25 06:54:11,447 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:11,536 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377933e4 to 127.0.0.1:60416 2023-04-25 06:54:11,536 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:11,539 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b41e306 to 127.0.0.1:60416 2023-04-25 06:54:11,539 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:11,539 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 06:54:11,539 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-25 06:54:11,539 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-04-25 06:54:11,539 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-25 06:54:11,539 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-25 06:54:11,539 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-04-25 06:54:11,539 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 3869 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 3818 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1676 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5026 rows 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1662 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 4986 rows 2023-04-25 06:54:11,540 INFO [Listener at localhost/41663] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 06:54:11,540 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36d9637e to 127.0.0.1:60416 2023-04-25 06:54:11,540 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:11,546 INFO [Listener at localhost/41663] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 06:54:11,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 06:54:11,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:11,561 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405651561"}]},"ts":"1682405651561"} 2023-04-25 06:54:11,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 06:54:11,563 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 06:54:11,564 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 06:54:11,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=266734ad02022a1c1fca6ab96809adb8, UNASSIGN}] 2023-04-25 06:54:11,568 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=266734ad02022a1c1fca6ab96809adb8, UNASSIGN 2023-04-25 06:54:11,568 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=266734ad02022a1c1fca6ab96809adb8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:11,569 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405651568"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405651568"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405651568"}]},"ts":"1682405651568"} 2023-04-25 06:54:11,570 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:54:11,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 06:54:11,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 266734ad02022a1c1fca6ab96809adb8, disabling compactions & flushes 2023-04-25 06:54:11,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. after waiting 0 ms 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:11,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 266734ad02022a1c1fca6ab96809adb8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=A 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=B 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 266734ad02022a1c1fca6ab96809adb8, store=C 2023-04-25 06:54:11,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:11,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 06:54:12,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/6fe3479674c443f68bd80911f771c35a 2023-04-25 06:54:12,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 06:54:12,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/efacaf1b30e24fa082556b67d8b28fbf 2023-04-25 06:54:12,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 06:54:12,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/17be5e5ac8624b51ba973d59299ceb60 2023-04-25 06:54:12,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/A/6fe3479674c443f68bd80911f771c35a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6fe3479674c443f68bd80911f771c35a 2023-04-25 06:54:12,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6fe3479674c443f68bd80911f771c35a, entries=50, sequenceid=506, filesize=7.1 K 2023-04-25 06:54:12,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/B/efacaf1b30e24fa082556b67d8b28fbf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/efacaf1b30e24fa082556b67d8b28fbf 2023-04-25 06:54:12,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/efacaf1b30e24fa082556b67d8b28fbf, entries=50, sequenceid=506, filesize=7.1 K 2023-04-25 06:54:12,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/.tmp/C/17be5e5ac8624b51ba973d59299ceb60 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/17be5e5ac8624b51ba973d59299ceb60 2023-04-25 06:54:12,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/17be5e5ac8624b51ba973d59299ceb60, entries=50, sequenceid=506, filesize=7.1 K 2023-04-25 06:54:12,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 266734ad02022a1c1fca6ab96809adb8 in 1254ms, sequenceid=506, compaction requested=true 2023-04-25 06:54:13,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b20f22b34ac949d882a8fb1b1d8c8c0a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b05e733f88bc40b0838ee32f8c48c084, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c72bbd08816a4971b307675ae808b38d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/df9a9ba8f1db411a934437e82e1032e1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/dacd8fee0aca4d2db8744b1bed8cf932, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c3b574468fee4bd49f720b53ab0a0f94, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/e01dfb3d2ec342e3a1426a282c9c1810, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/631def4486754da39c6b93479c98ea66, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/20512ec237e04b819f634f3cdb23f7a5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2c5ccc4957674ea1bec156476b0d3572, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5b0eda36652e48b284c82bedb32fa8ed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/a3839869210542edaa1ca8d9c56e7f6d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/94a90d34011c488fae1dd7810ad7b9c4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4d0e73e1081843c8a3917bde4d8453b3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9accb2d718e34f80a9173c2ee387324b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2af9708f3169496780ba99dec6d205d0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6be7818df43b42b18a1f86559f31d173, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/d5a8cec25a1b433e8107d7b1727f09e3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5e0335b5d14d40b19f021d50bb7b910d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/049039c5ee9e47f2ba65ca6a7977035d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9e6744dfc5b4422086f8421d276db39b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f489843f5cb2425f8578a115e715e985, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/76e2764d821c43e68b1e406bb8a2afd1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/0674064b435d4371ad8945358edfae62, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/ea85a7343b3f40bfbff6ce461a33f364, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/71ae4791bbb7489bbb7a3d6d54cae6ef, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3ab6f00bc62d49128bdd5f287159ca51, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2d0ab03f6ad64484a3ee06e3a4a48fd3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/246aeaaacd17443fb5c9be10805716b3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9ed770c8d1654394abf33cbb6286c1e4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4cf1ce40b90a4f2e89d2d3252a4610d1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/26e76b381d0a4a5f9f1d525e24042d23] to archive 2023-04-25 06:54:13,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:54:13,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b20f22b34ac949d882a8fb1b1d8c8c0a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b20f22b34ac949d882a8fb1b1d8c8c0a 2023-04-25 06:54:13,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b05e733f88bc40b0838ee32f8c48c084 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/b05e733f88bc40b0838ee32f8c48c084 2023-04-25 06:54:13,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c72bbd08816a4971b307675ae808b38d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c72bbd08816a4971b307675ae808b38d 2023-04-25 06:54:13,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/df9a9ba8f1db411a934437e82e1032e1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/df9a9ba8f1db411a934437e82e1032e1 2023-04-25 06:54:13,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/dacd8fee0aca4d2db8744b1bed8cf932 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/dacd8fee0aca4d2db8744b1bed8cf932 2023-04-25 06:54:13,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c3b574468fee4bd49f720b53ab0a0f94 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/c3b574468fee4bd49f720b53ab0a0f94 2023-04-25 06:54:13,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/e01dfb3d2ec342e3a1426a282c9c1810 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/e01dfb3d2ec342e3a1426a282c9c1810 2023-04-25 06:54:13,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/631def4486754da39c6b93479c98ea66 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/631def4486754da39c6b93479c98ea66 2023-04-25 06:54:13,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/20512ec237e04b819f634f3cdb23f7a5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/20512ec237e04b819f634f3cdb23f7a5 2023-04-25 06:54:13,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2c5ccc4957674ea1bec156476b0d3572 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2c5ccc4957674ea1bec156476b0d3572 2023-04-25 06:54:13,023 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5b0eda36652e48b284c82bedb32fa8ed to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5b0eda36652e48b284c82bedb32fa8ed 2023-04-25 06:54:13,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/a3839869210542edaa1ca8d9c56e7f6d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/a3839869210542edaa1ca8d9c56e7f6d 2023-04-25 06:54:13,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3b18bf8ab4704b3cb4ed2b6edb04d2bd 2023-04-25 06:54:13,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/94a90d34011c488fae1dd7810ad7b9c4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/94a90d34011c488fae1dd7810ad7b9c4 2023-04-25 06:54:13,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4d0e73e1081843c8a3917bde4d8453b3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4d0e73e1081843c8a3917bde4d8453b3 2023-04-25 06:54:13,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9accb2d718e34f80a9173c2ee387324b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9accb2d718e34f80a9173c2ee387324b 2023-04-25 06:54:13,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2af9708f3169496780ba99dec6d205d0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2af9708f3169496780ba99dec6d205d0 2023-04-25 06:54:13,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6be7818df43b42b18a1f86559f31d173 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6be7818df43b42b18a1f86559f31d173 2023-04-25 06:54:13,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/d5a8cec25a1b433e8107d7b1727f09e3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/d5a8cec25a1b433e8107d7b1727f09e3 2023-04-25 06:54:13,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5e0335b5d14d40b19f021d50bb7b910d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/5e0335b5d14d40b19f021d50bb7b910d 2023-04-25 06:54:13,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/049039c5ee9e47f2ba65ca6a7977035d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/049039c5ee9e47f2ba65ca6a7977035d 2023-04-25 06:54:13,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9e6744dfc5b4422086f8421d276db39b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9e6744dfc5b4422086f8421d276db39b 2023-04-25 06:54:13,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f489843f5cb2425f8578a115e715e985 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f489843f5cb2425f8578a115e715e985 2023-04-25 06:54:13,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/76e2764d821c43e68b1e406bb8a2afd1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/76e2764d821c43e68b1e406bb8a2afd1 2023-04-25 06:54:13,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/0674064b435d4371ad8945358edfae62 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/0674064b435d4371ad8945358edfae62 2023-04-25 06:54:13,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/ea85a7343b3f40bfbff6ce461a33f364 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/ea85a7343b3f40bfbff6ce461a33f364 2023-04-25 06:54:13,046 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/71ae4791bbb7489bbb7a3d6d54cae6ef to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/71ae4791bbb7489bbb7a3d6d54cae6ef 2023-04-25 06:54:13,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3ab6f00bc62d49128bdd5f287159ca51 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/3ab6f00bc62d49128bdd5f287159ca51 2023-04-25 06:54:13,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2d0ab03f6ad64484a3ee06e3a4a48fd3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/2d0ab03f6ad64484a3ee06e3a4a48fd3 2023-04-25 06:54:13,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/246aeaaacd17443fb5c9be10805716b3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/246aeaaacd17443fb5c9be10805716b3 2023-04-25 06:54:13,052 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9ed770c8d1654394abf33cbb6286c1e4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/9ed770c8d1654394abf33cbb6286c1e4 2023-04-25 06:54:13,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4cf1ce40b90a4f2e89d2d3252a4610d1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/4cf1ce40b90a4f2e89d2d3252a4610d1 2023-04-25 06:54:13,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/26e76b381d0a4a5f9f1d525e24042d23 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/26e76b381d0a4a5f9f1d525e24042d23 2023-04-25 06:54:13,081 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/52feb1bd68e7435e92bbf8e591fe4d09, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6e1bf8ccefb40dc99f778c31822c46d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0be88638532a47d4ad4d2c4b1b72a652, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/c9743864e82b4993aca4a7be4b778eec, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/2aa4ac39663f47c080d0a0b3eb7a4d72, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5b0d5709c8a2455f93961781c91c6421, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9069e3264dee4cf8b77f5df26daeedb2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0463cee3c5554a8fa0415fde23b321d3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/535fdcfc5cbd4018ae0dcb5dedee4d76, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e3ba237709c647c69d8e423554ae38e3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/46aa745efeab4b429f19cf1e4e9d8280, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9cfa964eeada4b6596c777d82c2f5424, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/58d759608e354b4db9de38e259ff337e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5381b5fa70f54cc39902bbf0b75d353e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/4631af755f30449c8361bf95bb00495b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bdadaba092042968b6d1f6150de55c2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/81358bc1036041139e3b42d8fd0c92c9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/eeceed4923a6463fb09ebbf6839a4b6f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/b4f9c071b0d749c19b3f41b7be4a79d9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/59ebf8224895460196016862a0b02fca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/daeea9611beb4ecba1a84feed985fd49, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/18ccd4a7ae4b4012a8e8736c628648ff, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bfada28234e45e8a1fa022748c17ef5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/22eec07ede3b43939d42131e85205704, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/3636701b058b48a58cf22bc2c7a4fb4d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e5f26c2581a24533806ed055e8421143, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6c3e6762e5741359b3268f1f3995146, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/13bd2139056e40c6b155f904781ec2e5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/03c728301a22450c950943ef32bf838f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/6675175fb9f544adb281fdb3821e9f1e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/948635f0373f4be7ad14d4abce0696d2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/7e6d9af2ed7f4225a0a7f7f3810898ff] to archive 2023-04-25 06:54:13,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:54:13,084 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/52feb1bd68e7435e92bbf8e591fe4d09 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/52feb1bd68e7435e92bbf8e591fe4d09 2023-04-25 06:54:13,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6e1bf8ccefb40dc99f778c31822c46d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6e1bf8ccefb40dc99f778c31822c46d 2023-04-25 06:54:13,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0be88638532a47d4ad4d2c4b1b72a652 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0be88638532a47d4ad4d2c4b1b72a652 2023-04-25 06:54:13,088 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/c9743864e82b4993aca4a7be4b778eec to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/c9743864e82b4993aca4a7be4b778eec 2023-04-25 06:54:13,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/2aa4ac39663f47c080d0a0b3eb7a4d72 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/2aa4ac39663f47c080d0a0b3eb7a4d72 2023-04-25 06:54:13,090 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5b0d5709c8a2455f93961781c91c6421 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5b0d5709c8a2455f93961781c91c6421 2023-04-25 06:54:13,092 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9069e3264dee4cf8b77f5df26daeedb2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9069e3264dee4cf8b77f5df26daeedb2 2023-04-25 06:54:13,093 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0463cee3c5554a8fa0415fde23b321d3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/0463cee3c5554a8fa0415fde23b321d3 2023-04-25 06:54:13,094 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/535fdcfc5cbd4018ae0dcb5dedee4d76 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/535fdcfc5cbd4018ae0dcb5dedee4d76 2023-04-25 06:54:13,095 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e3ba237709c647c69d8e423554ae38e3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e3ba237709c647c69d8e423554ae38e3 2023-04-25 06:54:13,097 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/46aa745efeab4b429f19cf1e4e9d8280 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/46aa745efeab4b429f19cf1e4e9d8280 2023-04-25 06:54:13,098 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9cfa964eeada4b6596c777d82c2f5424 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/9cfa964eeada4b6596c777d82c2f5424 2023-04-25 06:54:13,099 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/58d759608e354b4db9de38e259ff337e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/58d759608e354b4db9de38e259ff337e 2023-04-25 06:54:13,100 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5381b5fa70f54cc39902bbf0b75d353e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5381b5fa70f54cc39902bbf0b75d353e 2023-04-25 06:54:13,101 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/4631af755f30449c8361bf95bb00495b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/4631af755f30449c8361bf95bb00495b 2023-04-25 06:54:13,102 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bdadaba092042968b6d1f6150de55c2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bdadaba092042968b6d1f6150de55c2 2023-04-25 06:54:13,103 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/81358bc1036041139e3b42d8fd0c92c9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/81358bc1036041139e3b42d8fd0c92c9 2023-04-25 06:54:13,104 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/eeceed4923a6463fb09ebbf6839a4b6f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/eeceed4923a6463fb09ebbf6839a4b6f 2023-04-25 06:54:13,105 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/d7b42f5a9d1a41f0ba243a2b5de6ff1a 2023-04-25 06:54:13,106 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/b4f9c071b0d749c19b3f41b7be4a79d9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/b4f9c071b0d749c19b3f41b7be4a79d9 2023-04-25 06:54:13,107 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/59ebf8224895460196016862a0b02fca to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/59ebf8224895460196016862a0b02fca 2023-04-25 06:54:13,108 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/daeea9611beb4ecba1a84feed985fd49 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/daeea9611beb4ecba1a84feed985fd49 2023-04-25 06:54:13,109 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/18ccd4a7ae4b4012a8e8736c628648ff to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/18ccd4a7ae4b4012a8e8736c628648ff 2023-04-25 06:54:13,110 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bfada28234e45e8a1fa022748c17ef5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/5bfada28234e45e8a1fa022748c17ef5 2023-04-25 06:54:13,111 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/22eec07ede3b43939d42131e85205704 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/22eec07ede3b43939d42131e85205704 2023-04-25 06:54:13,112 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/3636701b058b48a58cf22bc2c7a4fb4d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/3636701b058b48a58cf22bc2c7a4fb4d 2023-04-25 06:54:13,113 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e5f26c2581a24533806ed055e8421143 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/e5f26c2581a24533806ed055e8421143 2023-04-25 06:54:13,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6c3e6762e5741359b3268f1f3995146 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/a6c3e6762e5741359b3268f1f3995146 2023-04-25 06:54:13,115 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/13bd2139056e40c6b155f904781ec2e5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/13bd2139056e40c6b155f904781ec2e5 2023-04-25 06:54:13,116 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/03c728301a22450c950943ef32bf838f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/03c728301a22450c950943ef32bf838f 2023-04-25 06:54:13,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/6675175fb9f544adb281fdb3821e9f1e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/6675175fb9f544adb281fdb3821e9f1e 2023-04-25 06:54:13,118 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/948635f0373f4be7ad14d4abce0696d2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/948635f0373f4be7ad14d4abce0696d2 2023-04-25 06:54:13,119 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/7e6d9af2ed7f4225a0a7f7f3810898ff to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/7e6d9af2ed7f4225a0a7f7f3810898ff 2023-04-25 06:54:13,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d32f56e4f20d46718bf670b834b71426, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3277b562c7154da894cbaa8c6fb07716, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/31f1c032f6bf403995f5bedb7c7b6568, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/13125f1e8cff43d5bb5555df00b7db8c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/747633842a354cdd91f152e0d2794aab, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/12d68387f9714cdc9ee518abbf1b70b4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/500f6f4cfb574704a54a5af8f966125f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/72a4fdf151e84475818b91506af8ddff, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/aea8a11944714b169c85008da5ecb96b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4b0f25723ed747dcbcf45d3a345c2e2a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1cc47876b2974ac282447b901dcefc09, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3498065cc3e540eab2e5ad16c9f3797f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9ff4edb3f91541569b7f87f1b6fb3406, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/016eb488528a4d929d7bf9f37f54ba97, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/6cbdbd8b091e4c48b9fed0e388daca66, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/2b574aaa22d9488483742ccb252b4c4e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/7ff60f9c0b95470ab6d6fd498d7584f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/5c403171c1cc4c14a8ec69c3ce762446, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d7687ccd70f04ab4823c4e22dd583e4d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a93718d73c3e45b59e3c2e1812219a16, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/b50aa104e77c49dba8ac93e30fc4a978, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9aa1796ad7a24e7a9345be7b6e5b7b1e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a6bab4ef600e4363a403245d9d6cb12c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/635082143bc34fdfb6143c4b505fb0e9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/509ad1e5ffc8437b8cea1045fdcb6edf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a261d2fa3d2f4c009e79283b3d294ed4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3a89a66db45448048c0eb862ba56f81c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/229f5f07fd744826a4123be0efc3524c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/32d57850a002411381ce40b87015a40f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/934a1745d4904c4b93efb3b2c2efe847, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/c35c50a47d87409f99952ba2ac83a97f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/650b90356f3742ef858ded0d25d4b3a6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1479d520ff1f402499946568f0f96f66] to archive 2023-04-25 06:54:13,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:54:13,125 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d32f56e4f20d46718bf670b834b71426 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d32f56e4f20d46718bf670b834b71426 2023-04-25 06:54:13,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3277b562c7154da894cbaa8c6fb07716 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3277b562c7154da894cbaa8c6fb07716 2023-04-25 06:54:13,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/31f1c032f6bf403995f5bedb7c7b6568 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/31f1c032f6bf403995f5bedb7c7b6568 2023-04-25 06:54:13,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/13125f1e8cff43d5bb5555df00b7db8c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/13125f1e8cff43d5bb5555df00b7db8c 2023-04-25 06:54:13,129 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/747633842a354cdd91f152e0d2794aab to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/747633842a354cdd91f152e0d2794aab 2023-04-25 06:54:13,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/12d68387f9714cdc9ee518abbf1b70b4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/12d68387f9714cdc9ee518abbf1b70b4 2023-04-25 06:54:13,132 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/500f6f4cfb574704a54a5af8f966125f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/500f6f4cfb574704a54a5af8f966125f 2023-04-25 06:54:13,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/72a4fdf151e84475818b91506af8ddff to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/72a4fdf151e84475818b91506af8ddff 2023-04-25 06:54:13,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/aea8a11944714b169c85008da5ecb96b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/aea8a11944714b169c85008da5ecb96b 2023-04-25 06:54:13,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4b0f25723ed747dcbcf45d3a345c2e2a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4b0f25723ed747dcbcf45d3a345c2e2a 2023-04-25 06:54:13,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1cc47876b2974ac282447b901dcefc09 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1cc47876b2974ac282447b901dcefc09 2023-04-25 06:54:13,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3498065cc3e540eab2e5ad16c9f3797f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3498065cc3e540eab2e5ad16c9f3797f 2023-04-25 06:54:13,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9ff4edb3f91541569b7f87f1b6fb3406 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9ff4edb3f91541569b7f87f1b6fb3406 2023-04-25 06:54:13,140 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/016eb488528a4d929d7bf9f37f54ba97 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/016eb488528a4d929d7bf9f37f54ba97 2023-04-25 06:54:13,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/6cbdbd8b091e4c48b9fed0e388daca66 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/6cbdbd8b091e4c48b9fed0e388daca66 2023-04-25 06:54:13,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/2b574aaa22d9488483742ccb252b4c4e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/2b574aaa22d9488483742ccb252b4c4e 2023-04-25 06:54:13,144 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/7ff60f9c0b95470ab6d6fd498d7584f7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/7ff60f9c0b95470ab6d6fd498d7584f7 2023-04-25 06:54:13,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/5c403171c1cc4c14a8ec69c3ce762446 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/5c403171c1cc4c14a8ec69c3ce762446 2023-04-25 06:54:13,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d7687ccd70f04ab4823c4e22dd583e4d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/d7687ccd70f04ab4823c4e22dd583e4d 2023-04-25 06:54:13,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a93718d73c3e45b59e3c2e1812219a16 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a93718d73c3e45b59e3c2e1812219a16 2023-04-25 06:54:13,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/b50aa104e77c49dba8ac93e30fc4a978 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/b50aa104e77c49dba8ac93e30fc4a978 2023-04-25 06:54:13,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9aa1796ad7a24e7a9345be7b6e5b7b1e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/9aa1796ad7a24e7a9345be7b6e5b7b1e 2023-04-25 06:54:13,150 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a6bab4ef600e4363a403245d9d6cb12c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a6bab4ef600e4363a403245d9d6cb12c 2023-04-25 06:54:13,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/635082143bc34fdfb6143c4b505fb0e9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/635082143bc34fdfb6143c4b505fb0e9 2023-04-25 06:54:13,153 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/509ad1e5ffc8437b8cea1045fdcb6edf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/509ad1e5ffc8437b8cea1045fdcb6edf 2023-04-25 06:54:13,154 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a261d2fa3d2f4c009e79283b3d294ed4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/a261d2fa3d2f4c009e79283b3d294ed4 2023-04-25 06:54:13,155 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3a89a66db45448048c0eb862ba56f81c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/3a89a66db45448048c0eb862ba56f81c 2023-04-25 06:54:13,156 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/229f5f07fd744826a4123be0efc3524c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/229f5f07fd744826a4123be0efc3524c 2023-04-25 06:54:13,157 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/32d57850a002411381ce40b87015a40f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/32d57850a002411381ce40b87015a40f 2023-04-25 06:54:13,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/934a1745d4904c4b93efb3b2c2efe847 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/934a1745d4904c4b93efb3b2c2efe847 2023-04-25 06:54:13,159 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/c35c50a47d87409f99952ba2ac83a97f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/c35c50a47d87409f99952ba2ac83a97f 2023-04-25 06:54:13,161 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/650b90356f3742ef858ded0d25d4b3a6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/650b90356f3742ef858ded0d25d4b3a6 2023-04-25 06:54:13,162 DEBUG [StoreCloser-TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1479d520ff1f402499946568f0f96f66 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1479d520ff1f402499946568f0f96f66 2023-04-25 06:54:13,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/recovered.edits/509.seqid, newMaxSeqId=509, maxSeqId=1 2023-04-25 06:54:13,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8. 2023-04-25 06:54:13,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 266734ad02022a1c1fca6ab96809adb8: 2023-04-25 06:54:13,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:13,171 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=266734ad02022a1c1fca6ab96809adb8, regionState=CLOSED 2023-04-25 06:54:13,171 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405653171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405653171"}]},"ts":"1682405653171"} 2023-04-25 06:54:13,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-25 06:54:13,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 266734ad02022a1c1fca6ab96809adb8, server=jenkins-hbase4.apache.org,41021,1682405622759 in 1.6020 sec 2023-04-25 06:54:13,176 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-25 06:54:13,176 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=266734ad02022a1c1fca6ab96809adb8, UNASSIGN in 1.6080 sec 2023-04-25 06:54:13,177 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405653177"}]},"ts":"1682405653177"} 2023-04-25 06:54:13,178 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 06:54:13,183 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 06:54:13,186 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6320 sec 2023-04-25 06:54:13,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 06:54:13,667 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-25 06:54:13,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 06:54:13,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:13,682 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:13,685 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:13,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 06:54:13,689 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:13,692 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/recovered.edits] 2023-04-25 06:54:13,695 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/229cf56ec9a7488d9d0a867d555e888a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/229cf56ec9a7488d9d0a867d555e888a 2023-04-25 06:54:13,696 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6fe3479674c443f68bd80911f771c35a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/6fe3479674c443f68bd80911f771c35a 2023-04-25 06:54:13,698 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f146d982316d41baba5ea5fcb44db23f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/A/f146d982316d41baba5ea5fcb44db23f 2023-04-25 06:54:13,700 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/36f7b2dda1d14f29bad7c775c688bdf3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/36f7b2dda1d14f29bad7c775c688bdf3 2023-04-25 06:54:13,702 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/ed08a12e3b9d4d208ddaa208ab216726 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/ed08a12e3b9d4d208ddaa208ab216726 2023-04-25 06:54:13,703 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/efacaf1b30e24fa082556b67d8b28fbf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/B/efacaf1b30e24fa082556b67d8b28fbf 2023-04-25 06:54:13,706 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/17be5e5ac8624b51ba973d59299ceb60 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/17be5e5ac8624b51ba973d59299ceb60 2023-04-25 06:54:13,707 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1d482d179762440bb34e25e76beabdd5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/1d482d179762440bb34e25e76beabdd5 2023-04-25 06:54:13,708 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4683f3a6bc864b00b71bed06baca0c27 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/C/4683f3a6bc864b00b71bed06baca0c27 2023-04-25 06:54:13,711 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/recovered.edits/509.seqid to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8/recovered.edits/509.seqid 2023-04-25 06:54:13,712 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/266734ad02022a1c1fca6ab96809adb8 2023-04-25 06:54:13,712 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:54:13,714 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:13,742 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 06:54:13,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 06:54:13,796 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 06:54:13,798 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:13,798 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 06:54:13,798 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682405653798"}]},"ts":"9223372036854775807"} 2023-04-25 06:54:13,801 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 06:54:13,801 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 266734ad02022a1c1fca6ab96809adb8, NAME => 'TestAcidGuarantees,,1682405626109.266734ad02022a1c1fca6ab96809adb8.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 06:54:13,801 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 06:54:13,802 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682405653801"}]},"ts":"9223372036854775807"} 2023-04-25 06:54:13,804 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 06:54:13,806 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:13,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 132 msec 2023-04-25 06:54:13,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 06:54:13,989 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-25 06:54:14,006 INFO [Listener at localhost/41663] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2781da71-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1953837160_15 at /127.0.0.1:57650 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1953837160_15 at /127.0.0.1:57592 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:41021-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2781da71-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2781da71-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2781da71-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=561 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=386 (was 315) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=4382 (was 5030) 2023-04-25 06:54:14,017 INFO [Listener at localhost/41663] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=172, AvailableMemoryMB=4382 2023-04-25 06:54:14,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:54:14,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:54:14,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:14,023 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 06:54:14,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-25 06:54:14,023 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 06:54:14,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 06:54:14,025 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:14,026 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 empty. 2023-04-25 06:54:14,026 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:14,026 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:54:14,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 06:54:14,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 06:54:14,435 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:54:14,437 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => de84d6c21b0ed9e7c77166f8a0775f87, NAME => 'TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp 2023-04-25 06:54:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 06:54:14,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:54:14,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing de84d6c21b0ed9e7c77166f8a0775f87, disabling compactions & flushes 2023-04-25 06:54:14,846 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:14,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:14,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. after waiting 0 ms 2023-04-25 06:54:14,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:14,846 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:14,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:14,849 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 06:54:14,850 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405654850"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405654850"}]},"ts":"1682405654850"} 2023-04-25 06:54:14,852 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 06:54:14,853 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 06:54:14,853 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405654853"}]},"ts":"1682405654853"} 2023-04-25 06:54:14,854 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 06:54:14,858 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, ASSIGN}] 2023-04-25 06:54:14,860 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, ASSIGN 2023-04-25 06:54:14,860 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=false 2023-04-25 06:54:15,012 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:15,012 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405655012"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405655012"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405655012"}]},"ts":"1682405655012"} 2023-04-25 06:54:15,014 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:54:15,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 06:54:15,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:15,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => de84d6c21b0ed9e7c77166f8a0775f87, NAME => 'TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:54:15,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:54:15,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,172 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,173 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A 2023-04-25 06:54:15,173 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A 2023-04-25 06:54:15,174 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:15,174 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de84d6c21b0ed9e7c77166f8a0775f87 columnFamilyName A 2023-04-25 06:54:15,174 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(310): Store=de84d6c21b0ed9e7c77166f8a0775f87/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:15,175 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,176 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B 2023-04-25 06:54:15,176 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B 2023-04-25 06:54:15,176 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:15,176 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de84d6c21b0ed9e7c77166f8a0775f87 columnFamilyName B 2023-04-25 06:54:15,177 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(310): Store=de84d6c21b0ed9e7c77166f8a0775f87/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:15,177 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,178 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C 2023-04-25 06:54:15,178 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C 2023-04-25 06:54:15,178 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:15,178 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de84d6c21b0ed9e7c77166f8a0775f87 columnFamilyName C 2023-04-25 06:54:15,179 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(310): Store=de84d6c21b0ed9e7c77166f8a0775f87/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:15,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:15,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:54:15,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:15,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:54:15,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened de84d6c21b0ed9e7c77166f8a0775f87; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72930870, jitterRate=0.08675464987754822}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:54:15,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:15,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., pid=18, masterSystemTime=1682405655166 2023-04-25 06:54:15,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:15,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:15,190 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:15,191 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405655190"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405655190"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405655190"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405655190"}]},"ts":"1682405655190"} 2023-04-25 06:54:15,194 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-25 06:54:15,194 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 in 178 msec 2023-04-25 06:54:15,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-25 06:54:15,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, ASSIGN in 337 msec 2023-04-25 06:54:15,196 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 06:54:15,196 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405655196"}]},"ts":"1682405655196"} 2023-04-25 06:54:15,197 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 06:54:15,199 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 06:54:15,201 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1800 sec 2023-04-25 06:54:16,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 06:54:16,130 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-25 06:54:16,131 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x478a542a to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:16,137 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a1406b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:16,138 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:16,151 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:16,153 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 06:54:16,156 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52452, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 06:54:16,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:54:16,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:54:16,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:16,594 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 06:54:16,595 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:54:16,595 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 06:54:16,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 06:54:16,615 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, REOPEN/MOVE}] 2023-04-25 06:54:16,616 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, REOPEN/MOVE 2023-04-25 06:54:16,617 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:16,617 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405656617"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405656617"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405656617"}]},"ts":"1682405656617"} 2023-04-25 06:54:16,619 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:54:16,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:16,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing de84d6c21b0ed9e7c77166f8a0775f87, disabling compactions & flushes 2023-04-25 06:54:16,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:16,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:16,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. after waiting 0 ms 2023-04-25 06:54:16,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:16,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 06:54:16,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:16,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:16,777 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: de84d6c21b0ed9e7c77166f8a0775f87 to self. 2023-04-25 06:54:16,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:16,779 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=CLOSED 2023-04-25 06:54:16,779 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405656779"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405656779"}]},"ts":"1682405656779"} 2023-04-25 06:54:16,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-25 06:54:16,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 in 162 msec 2023-04-25 06:54:16,782 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=true 2023-04-25 06:54:16,932 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:16,933 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405656932"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405656932"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405656932"}]},"ts":"1682405656932"} 2023-04-25 06:54:16,934 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:54:17,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => de84d6c21b0ed9e7c77166f8a0775f87, NAME => 'TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:54:17,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:54:17,091 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,091 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,093 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,094 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A 2023-04-25 06:54:17,094 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A 2023-04-25 06:54:17,094 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:17,097 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de84d6c21b0ed9e7c77166f8a0775f87 columnFamilyName A 2023-04-25 06:54:17,099 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(310): Store=de84d6c21b0ed9e7c77166f8a0775f87/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:17,099 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,100 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B 2023-04-25 06:54:17,100 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B 2023-04-25 06:54:17,101 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:17,101 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de84d6c21b0ed9e7c77166f8a0775f87 columnFamilyName B 2023-04-25 06:54:17,102 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(310): Store=de84d6c21b0ed9e7c77166f8a0775f87/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:17,102 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,102 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C 2023-04-25 06:54:17,102 DEBUG [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C 2023-04-25 06:54:17,103 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:17,103 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de84d6c21b0ed9e7c77166f8a0775f87 columnFamilyName C 2023-04-25 06:54:17,104 INFO [StoreOpener-de84d6c21b0ed9e7c77166f8a0775f87-1] regionserver.HStore(310): Store=de84d6c21b0ed9e7c77166f8a0775f87/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:17,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:54:17,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened de84d6c21b0ed9e7c77166f8a0775f87; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66989447, jitterRate=-0.0017794519662857056}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:54:17,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:17,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., pid=23, masterSystemTime=1682405657086 2023-04-25 06:54:17,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,112 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,112 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405657112"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405657112"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405657112"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405657112"}]},"ts":"1682405657112"} 2023-04-25 06:54:17,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-25 06:54:17,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 in 180 msec 2023-04-25 06:54:17,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-25 06:54:17,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, REOPEN/MOVE in 500 msec 2023-04-25 06:54:17,120 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-25 06:54:17,120 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 519 msec 2023-04-25 06:54:17,122 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 951 msec 2023-04-25 06:54:17,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-25 06:54:17,131 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x542e33ab to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,136 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c508332, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,137 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x46078b1e to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,141 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7adf5e8e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,141 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x28fcbe97 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,146 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52d98fed, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,147 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x18496e17 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,151 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f13c14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,152 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x76aef981 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,155 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25441895, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,156 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x40e0a3dc to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,160 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45e59be5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,160 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x63d375f8 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,165 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2668d41d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,165 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x1345b761 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,169 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b416c44, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,169 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x1e0b0a9f to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:17,174 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@204b5880, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:17,180 DEBUG [hconnection-0x4f1c6982-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,181 DEBUG [hconnection-0x5dcd6e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,181 DEBUG [hconnection-0x7a22c5c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:17,182 DEBUG [hconnection-0x62497345-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,183 DEBUG [hconnection-0x53b42b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,183 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,183 DEBUG [hconnection-0x44a2174c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,183 DEBUG [hconnection-0x5c547a65-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,184 DEBUG [hconnection-0x4fedaced-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,184 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,184 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,189 DEBUG [hconnection-0x61688c4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:17,189 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,189 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,190 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,190 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,191 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51406, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,191 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:17,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:17,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:17,225 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:17,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:17,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:17,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:17,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:54:17,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:17,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:17,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:17,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:17,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:17,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:17,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:17,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:17,228 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,228 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405717242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405717244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405717248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405717245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405717248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:17,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:17,268 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:17,268 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:17,270 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,270 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:17,270 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:17,270 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:17,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58363b90[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:17,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:17,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,279 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:17,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:17,279 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:17,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:17,282 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:17,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:17,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:17,285 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:17,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:17,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:17,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:17,285 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:17,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:17,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,288 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,291 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:17,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:17,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:17,295 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:17,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:17,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:17,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:17,296 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:17,296 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:17,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405717347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,351 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304250e6a392c358c4c078917f7fbfd7b9dbe to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250e6a392c358c4c078917f7fbfd7b9dbe 2023-04-25 06:54:17,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405717351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405717351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405717354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405717355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,396 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:17,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:17,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:17,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:17,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:17,424 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:17,424 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:17,424 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:17,424 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:17,424 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/41456a0a927041e48437b7aaccf85e1c 2023-04-25 06:54:17,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:17,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:17,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:17,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:17,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:17,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:17,448 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:17,448 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:17,448 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:17,460 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,460 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:17,460 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:17,460 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:17,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:17,478 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:17,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:17,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:17,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f944d34[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:17,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:17,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,485 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:17,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:17,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:17,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:17,488 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:17,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:17,497 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:17,497 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:17,497 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:17,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/b6e044b6347b43cfaa3ba1b5c5ac6530 2023-04-25 06:54:17,497 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:17,498 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:17,500 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,500 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:17,500 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:17,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:17,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:17,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:17,520 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:17,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:17,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:17,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,520 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,520 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,520 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,520 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:17,520 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,520 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:17,520 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:17,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:17,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405717553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405717554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405717558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405717563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/f4a56b446cb242d092af769979574806 2023-04-25 06:54:17,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405717563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/41456a0a927041e48437b7aaccf85e1c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/41456a0a927041e48437b7aaccf85e1c 2023-04-25 06:54:17,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/41456a0a927041e48437b7aaccf85e1c, entries=150, sequenceid=18, filesize=25.1 K 2023-04-25 06:54:17,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/b6e044b6347b43cfaa3ba1b5c5ac6530 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b6e044b6347b43cfaa3ba1b5c5ac6530 2023-04-25 06:54:17,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b6e044b6347b43cfaa3ba1b5c5ac6530, entries=100, sequenceid=18, filesize=9.3 K 2023-04-25 06:54:17,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/f4a56b446cb242d092af769979574806 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/f4a56b446cb242d092af769979574806 2023-04-25 06:54:17,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/f4a56b446cb242d092af769979574806, entries=100, sequenceid=18, filesize=9.3 K 2023-04-25 06:54:17,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for de84d6c21b0ed9e7c77166f8a0775f87 in 388ms, sequenceid=18, compaction requested=false 2023-04-25 06:54:17,613 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 06:54:17,615 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:17,620 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:17,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:17,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:17,627 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:17,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:17,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:17,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:17,628 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,630 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:17,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:17,630 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:17,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:17,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:17,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:17,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:17,640 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:17,640 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:17,640 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:17,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:17,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:17,658 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:17,658 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:17,658 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:54:17,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:17,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:17,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:17,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:17,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:17,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:17,713 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425daf51c83f2dd406cbb10486571f4af25 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425daf51c83f2dd406cbb10486571f4af25 2023-04-25 06:54:17,752 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/32cd490b558646dd90e9cb02f617258d 2023-04-25 06:54:17,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:17,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:17,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405717871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405717872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405717873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405717874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405717879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405717980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405717980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405717981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405717982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:17,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:17,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405717984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405718185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405718186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405718186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405718187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405718187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,221 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/12079431d8314b68a4b587d481cca8b6 2023-04-25 06:54:18,254 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/728c22f1fda843cb8bb7ac24a77b6355 2023-04-25 06:54:18,260 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/32cd490b558646dd90e9cb02f617258d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/32cd490b558646dd90e9cb02f617258d 2023-04-25 06:54:18,267 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/32cd490b558646dd90e9cb02f617258d, entries=150, sequenceid=37, filesize=25.1 K 2023-04-25 06:54:18,269 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/12079431d8314b68a4b587d481cca8b6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/12079431d8314b68a4b587d481cca8b6 2023-04-25 06:54:18,278 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/12079431d8314b68a4b587d481cca8b6, entries=150, sequenceid=37, filesize=11.6 K 2023-04-25 06:54:18,279 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/728c22f1fda843cb8bb7ac24a77b6355 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/728c22f1fda843cb8bb7ac24a77b6355 2023-04-25 06:54:18,285 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/728c22f1fda843cb8bb7ac24a77b6355, entries=150, sequenceid=37, filesize=11.6 K 2023-04-25 06:54:18,287 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for de84d6c21b0ed9e7c77166f8a0775f87 in 628ms, sequenceid=37, compaction requested=false 2023-04-25 06:54:18,287 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:18,287 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:18,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:18,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:18,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:18,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a71359a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:18,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:18,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:18,294 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,294 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:18,299 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:18,303 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:18,303 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:18,303 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:18,303 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:18,303 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:18,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:18,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:18,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:18,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:18,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,311 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:18,311 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:18,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:18,311 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:18,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:18,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:18,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:18,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:18,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:18,312 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:18,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:18,312 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:18,312 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:18,412 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:18,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:18,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:18,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:18,425 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:18,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:18,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:18,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:18,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:18,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,428 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:18,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:18,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:18,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:18,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:18,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:18,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:18,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:18,440 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:18,440 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:18,440 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:54:18,441 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:18,441 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:18,441 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:18,441 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:18,441 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:18,441 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:18,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:18,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:18,493 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254884f98d886a46069eca2be5f5cdb2d6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254884f98d886a46069eca2be5f5cdb2d6 2023-04-25 06:54:18,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405718512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405718515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405718516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405718520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405718520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405718619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405718621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405718624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405718626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405718626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,742 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 06:54:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405718824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405718826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405718828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:18,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405718834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405718835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:18,901 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/beaaf686aecd419195a83db5f17a3e7e 2023-04-25 06:54:19,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405719127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405719129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405719130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405719140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405719143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,326 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/8d15340ac00b4d42826ae79dc4f78236 2023-04-25 06:54:19,341 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/acadb9396f3043b0849335155930a213 2023-04-25 06:54:19,346 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/beaaf686aecd419195a83db5f17a3e7e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/beaaf686aecd419195a83db5f17a3e7e 2023-04-25 06:54:19,351 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/beaaf686aecd419195a83db5f17a3e7e, entries=150, sequenceid=50, filesize=25.1 K 2023-04-25 06:54:19,352 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/8d15340ac00b4d42826ae79dc4f78236 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8d15340ac00b4d42826ae79dc4f78236 2023-04-25 06:54:19,358 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8d15340ac00b4d42826ae79dc4f78236, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 06:54:19,359 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/acadb9396f3043b0849335155930a213 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/acadb9396f3043b0849335155930a213 2023-04-25 06:54:19,364 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/acadb9396f3043b0849335155930a213, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 06:54:19,365 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for de84d6c21b0ed9e7c77166f8a0775f87 in 925ms, sequenceid=50, compaction requested=true 2023-04-25 06:54:19,365 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:19,365 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:19,369 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d8bad0b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:19,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:19,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,375 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:19,376 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,376 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,377 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:19,381 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:19,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:19,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:19,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:19,382 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:19,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,387 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:19,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:19,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:19,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:19,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:19,395 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:19,396 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:19,496 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:19,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:19,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:19,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:19,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:19,502 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:19,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:19,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:19,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:19,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,504 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:19,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:19,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:19,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:19,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:19,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:19,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:19,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:19,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:19,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:19,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:19,516 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:19,516 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:19,517 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:54:19,517 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:19,517 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:19,517 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:19,517 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:19,517 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:19,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:19,570 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425cab2cdc32abd46bf9647384c7e2eca35 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cab2cdc32abd46bf9647384c7e2eca35 2023-04-25 06:54:19,585 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/e21d7d11617d47d59a4596e7533e914d 2023-04-25 06:54:19,618 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/99cfc7ab27574c58bf3cee635ee85a34 2023-04-25 06:54:19,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:19,634 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:19,650 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/e1c9e5279ae243cba568b31771707d92 2023-04-25 06:54:19,655 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/e21d7d11617d47d59a4596e7533e914d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/e21d7d11617d47d59a4596e7533e914d 2023-04-25 06:54:19,663 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/e21d7d11617d47d59a4596e7533e914d, entries=150, sequenceid=70, filesize=25.1 K 2023-04-25 06:54:19,664 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/99cfc7ab27574c58bf3cee635ee85a34 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/99cfc7ab27574c58bf3cee635ee85a34 2023-04-25 06:54:19,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405719660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405719666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,672 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/99cfc7ab27574c58bf3cee635ee85a34, entries=150, sequenceid=70, filesize=11.6 K 2023-04-25 06:54:19,673 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/e1c9e5279ae243cba568b31771707d92 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e1c9e5279ae243cba568b31771707d92 2023-04-25 06:54:19,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405719670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405719672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405719672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,680 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e1c9e5279ae243cba568b31771707d92, entries=150, sequenceid=70, filesize=11.6 K 2023-04-25 06:54:19,681 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for de84d6c21b0ed9e7c77166f8a0775f87 in 164ms, sequenceid=70, compaction requested=true 2023-04-25 06:54:19,681 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:19,681 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:19,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2488e591[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:19,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:19,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:19,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:19,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:19,693 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:19,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:19,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:19,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:19,694 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:19,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:19,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,702 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:19,702 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,702 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:19,702 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:19,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:19,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:19,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:19,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:54:19,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:19,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:19,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:19,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:19,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:19,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:19,803 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:19,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:19,808 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425589ca9d6bbf54a3596a3aed494c6534a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425589ca9d6bbf54a3596a3aed494c6534a 2023-04-25 06:54:19,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:19,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:19,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:19,813 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:19,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:19,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:19,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:19,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,815 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:19,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:19,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405719810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405719812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:19,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405719812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405719818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405719819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:19,820 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:19,820 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:19,820 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:19,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:19,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:19,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:19,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:19,830 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:19,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5908f1d9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:19,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:19,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:19,836 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:19,838 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:19,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:19,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:19,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:19,838 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:19,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,845 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,848 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:19,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:19,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:19,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,849 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:19,849 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:19,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:19,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405719917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405719917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405719919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405719921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405719921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,949 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:19,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:19,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:19,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:19,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:19,953 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:19,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:19,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:19,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:19,954 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,954 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:19,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:19,957 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:19,957 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:19,957 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:19,957 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,959 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a76c287[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:19,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:19,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,962 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:19,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:19,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:19,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:19,969 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:19,969 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:19,969 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:19,969 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:19,969 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:19,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:19,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,972 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:19,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:19,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:19,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:19,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:19,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:19,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:19,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:19,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:19,978 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:19,978 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:19,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:19,979 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,979 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:19,979 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:19,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,079 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:20,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:20,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:20,082 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:20,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:20,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:20,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:20,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:20,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:20,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:20,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:20,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:20,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,088 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a05cb37[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:20,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:20,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:20,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:20,093 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:20,096 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:20,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:20,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:20,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:20,096 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:20,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,098 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:20,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:20,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:20,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,103 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:20,104 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,104 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,104 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405720122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405720122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405720123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405720127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405720128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,204 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:20,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:20,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:20,208 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:20,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:20,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:20,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,210 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:20,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:20,217 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:20,217 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:20,218 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9c20f8b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:20,222 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:20,222 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,224 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:20,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:20,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:20,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:20,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:20,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:20,227 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/a0e368d290b4419796920d1370f87acf 2023-04-25 06:54:20,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:20,233 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:20,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:20,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:20,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:20,233 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:20,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,238 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,244 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,244 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,244 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,244 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,244 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:20,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:20,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:20,244 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,245 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,245 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,245 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,245 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:20,245 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:20,345 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:20,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:20,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:20,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:20,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:20,349 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:20,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:20,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:20,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,351 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,351 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,351 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:20,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:20,353 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,353 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:20,353 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:20,353 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:20,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@175ab153[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:20,357 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:20,357 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,359 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,359 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:20,359 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:20,359 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:20,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:20,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:20,364 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:20,364 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:20,364 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:20,364 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:20,364 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:20,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,367 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,370 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,371 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:20,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,371 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:20,371 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:20,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405720426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405720427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405720427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405720432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405720432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,471 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:20,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:20,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:20,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:20,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:20,475 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:20,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:20,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:20,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,477 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,478 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:20,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:20,479 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,479 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:20,479 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:20,479 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:20,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b742bc5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:20,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:20,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,485 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:20,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:20,489 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:20,491 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:20,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:20,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:20,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:20,491 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:20,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,494 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,495 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,496 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:20,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:20,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,499 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:20,500 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:20,600 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:20,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:20,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:20,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:20,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:20,603 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:20,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:20,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:20,604 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,604 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,605 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,605 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,606 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,606 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:20,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:20,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:20,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:20,607 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:20,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:20,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6c0fef[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:20,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:20,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,616 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:20,616 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,616 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:20,617 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:20,621 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:20,621 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:20,621 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:20,621 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:20,621 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:20,623 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,623 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,623 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,624 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:20,626 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:20,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:20,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:20,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:20,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:20,633 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:20,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,633 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:20,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:20,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/d7cfa5b1ad4345a08749fa38953cba5a 2023-04-25 06:54:20,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/e8ef650e5f8e47b99f463189d6c79aaa 2023-04-25 06:54:20,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/a0e368d290b4419796920d1370f87acf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a0e368d290b4419796920d1370f87acf 2023-04-25 06:54:20,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a0e368d290b4419796920d1370f87acf, entries=250, sequenceid=85, filesize=38.7 K 2023-04-25 06:54:20,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/d7cfa5b1ad4345a08749fa38953cba5a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/d7cfa5b1ad4345a08749fa38953cba5a 2023-04-25 06:54:20,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/d7cfa5b1ad4345a08749fa38953cba5a, entries=150, sequenceid=85, filesize=11.6 K 2023-04-25 06:54:20,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/e8ef650e5f8e47b99f463189d6c79aaa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e8ef650e5f8e47b99f463189d6c79aaa 2023-04-25 06:54:20,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e8ef650e5f8e47b99f463189d6c79aaa, entries=150, sequenceid=85, filesize=11.6 K 2023-04-25 06:54:20,733 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for de84d6c21b0ed9e7c77166f8a0775f87 in 957ms, sequenceid=85, compaction requested=true 2023-04-25 06:54:20,733 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:20,733 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:20,733 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:20,736 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:20,736 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:20,735 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:20,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:20,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:20,742 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 142610 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:20,742 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/A is initiating minor compaction (all files) 2023-04-25 06:54:20,742 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/A in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,742 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:20,742 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/B is initiating minor compaction (all files) 2023-04-25 06:54:20,742 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/41456a0a927041e48437b7aaccf85e1c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/32cd490b558646dd90e9cb02f617258d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/beaaf686aecd419195a83db5f17a3e7e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/e21d7d11617d47d59a4596e7533e914d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a0e368d290b4419796920d1370f87acf] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=139.3 K 2023-04-25 06:54:20,742 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/B in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,742 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b6e044b6347b43cfaa3ba1b5c5ac6530, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/12079431d8314b68a4b587d481cca8b6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8d15340ac00b4d42826ae79dc4f78236, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/99cfc7ab27574c58bf3cee635ee85a34, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/d7cfa5b1ad4345a08749fa38953cba5a] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=55.7 K 2023-04-25 06:54:20,743 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 41456a0a927041e48437b7aaccf85e1c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682405657209 2023-04-25 06:54:20,743 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b6e044b6347b43cfaa3ba1b5c5ac6530, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682405657225 2023-04-25 06:54:20,743 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 32cd490b558646dd90e9cb02f617258d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682405657241 2023-04-25 06:54:20,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:20,743 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 12079431d8314b68a4b587d481cca8b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682405657241 2023-04-25 06:54:20,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:20,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:20,745 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:20,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:20,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:20,745 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting beaaf686aecd419195a83db5f17a3e7e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405657870 2023-04-25 06:54:20,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,745 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 8d15340ac00b4d42826ae79dc4f78236, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405657870 2023-04-25 06:54:20,746 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting e21d7d11617d47d59a4596e7533e914d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682405658509 2023-04-25 06:54:20,746 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 99cfc7ab27574c58bf3cee635ee85a34, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682405658509 2023-04-25 06:54:20,746 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a0e368d290b4419796920d1370f87acf, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405659665 2023-04-25 06:54:20,746 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting d7cfa5b1ad4345a08749fa38953cba5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405659669 2023-04-25 06:54:20,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:20,747 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:20,747 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,747 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:20,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:20,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:20,749 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,750 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:20,750 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:54:20,750 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:20,750 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:20,750 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:20,751 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:20,751 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:20,751 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:20,765 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#B#compaction#124 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:20,785 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#A#compaction#123 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:20,813 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/dfdb2b8600eb4872ac614a00f29c8b7b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/dfdb2b8600eb4872ac614a00f29c8b7b 2023-04-25 06:54:20,818 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255723bb0a1c03467bab5133472e5b95dd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255723bb0a1c03467bab5133472e5b95dd 2023-04-25 06:54:20,822 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/B of de84d6c21b0ed9e7c77166f8a0775f87 into dfdb2b8600eb4872ac614a00f29c8b7b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:20,822 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:20,822 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/B, priority=11, startTime=1682405660733; duration=0sec 2023-04-25 06:54:20,822 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:20,822 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:20,824 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:20,824 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/C is initiating minor compaction (all files) 2023-04-25 06:54:20,824 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/C in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:20,824 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/f4a56b446cb242d092af769979574806, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/728c22f1fda843cb8bb7ac24a77b6355, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/acadb9396f3043b0849335155930a213, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e1c9e5279ae243cba568b31771707d92, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e8ef650e5f8e47b99f463189d6c79aaa] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=55.7 K 2023-04-25 06:54:20,825 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting f4a56b446cb242d092af769979574806, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682405657225 2023-04-25 06:54:20,826 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 728c22f1fda843cb8bb7ac24a77b6355, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682405657241 2023-04-25 06:54:20,826 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting acadb9396f3043b0849335155930a213, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405657870 2023-04-25 06:54:20,828 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e1c9e5279ae243cba568b31771707d92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682405658509 2023-04-25 06:54:20,829 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/f09578c3d7dd43948975526254778a08 2023-04-25 06:54:20,835 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e8ef650e5f8e47b99f463189d6c79aaa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405659669 2023-04-25 06:54:20,854 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#C#compaction#127 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:20,858 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/322184d7ed4f42f7921f321229e526aa 2023-04-25 06:54:20,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:20,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:20,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405720944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405720946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405720969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405720969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:20,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405720969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405721070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405721070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405721075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405721082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405721082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405721275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405721275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,281 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/93249af5659643c98fc52d4396dbc9dc as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/93249af5659643c98fc52d4396dbc9dc 2023-04-25 06:54:21,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405721280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,285 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/fc2f49cbd0ee43d7bee804672b03051f 2023-04-25 06:54:21,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405721287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405721287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,290 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/f09578c3d7dd43948975526254778a08 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f09578c3d7dd43948975526254778a08 2023-04-25 06:54:21,291 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/C of de84d6c21b0ed9e7c77166f8a0775f87 into 93249af5659643c98fc52d4396dbc9dc(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:21,291 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:21,292 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/C, priority=11, startTime=1682405660736; duration=0sec 2023-04-25 06:54:21,292 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:21,296 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f09578c3d7dd43948975526254778a08, entries=150, sequenceid=102, filesize=25.1 K 2023-04-25 06:54:21,297 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/322184d7ed4f42f7921f321229e526aa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/322184d7ed4f42f7921f321229e526aa 2023-04-25 06:54:21,303 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/322184d7ed4f42f7921f321229e526aa, entries=150, sequenceid=102, filesize=11.6 K 2023-04-25 06:54:21,311 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/fc2f49cbd0ee43d7bee804672b03051f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fc2f49cbd0ee43d7bee804672b03051f 2023-04-25 06:54:21,318 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fc2f49cbd0ee43d7bee804672b03051f, entries=150, sequenceid=102, filesize=11.6 K 2023-04-25 06:54:21,319 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=93.93 KB/96180 for de84d6c21b0ed9e7c77166f8a0775f87 in 569ms, sequenceid=102, compaction requested=false 2023-04-25 06:54:21,319 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:21,319 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:21,322 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:21,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:21,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cbe5de[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:21,324 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:21,324 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,328 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:21,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:21,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:21,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:21,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:21,335 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:21,335 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:21,335 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:21,335 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:21,335 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:21,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:21,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:21,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:21,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:21,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:21,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:21,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:21,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:21,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:21,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:21,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:21,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,347 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:21,347 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:21,447 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:21,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:21,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:21,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:21,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:21,451 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:21,451 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:21,451 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:21,451 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:21,452 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,452 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:21,454 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:21,454 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:21,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:21,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:21,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:21,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:21,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:21,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:21,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:21,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:21,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:21,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:21,456 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:21,456 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:21,457 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:54:21,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:21,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:21,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:21,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:21,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:21,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:21,484 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257692501dff26441bb1505abf1db43fbd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257692501dff26441bb1505abf1db43fbd 2023-04-25 06:54:21,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:21,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:21,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405721593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405721594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405721596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405721596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405721597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,627 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/5df193c518054c16a09e9f5492eb9c36 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5df193c518054c16a09e9f5492eb9c36 2023-04-25 06:54:21,632 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/A of de84d6c21b0ed9e7c77166f8a0775f87 into 5df193c518054c16a09e9f5492eb9c36(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:21,632 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:21,632 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/A, priority=11, startTime=1682405660733; duration=0sec 2023-04-25 06:54:21,633 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:21,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405721697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405721698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405721699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405721700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405721701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405721900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405721900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405721901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,902 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=121, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/69406afb6e5d47fc8526170ae2c98670 2023-04-25 06:54:21,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405721905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:21,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405721905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:21,944 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/b64deabbd0b643d8a0a963b553676b9f 2023-04-25 06:54:22,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405722202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405722204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405722205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405722217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405722219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,362 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/a9b3d972d2d44af28a82452447c6fbbc 2023-04-25 06:54:22,368 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/69406afb6e5d47fc8526170ae2c98670 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/69406afb6e5d47fc8526170ae2c98670 2023-04-25 06:54:22,376 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/69406afb6e5d47fc8526170ae2c98670, entries=150, sequenceid=121, filesize=25.1 K 2023-04-25 06:54:22,377 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/b64deabbd0b643d8a0a963b553676b9f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b64deabbd0b643d8a0a963b553676b9f 2023-04-25 06:54:22,382 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b64deabbd0b643d8a0a963b553676b9f, entries=150, sequenceid=121, filesize=11.6 K 2023-04-25 06:54:22,383 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/a9b3d972d2d44af28a82452447c6fbbc as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9b3d972d2d44af28a82452447c6fbbc 2023-04-25 06:54:22,388 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9b3d972d2d44af28a82452447c6fbbc, entries=150, sequenceid=121, filesize=11.6 K 2023-04-25 06:54:22,389 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for de84d6c21b0ed9e7c77166f8a0775f87 in 932ms, sequenceid=121, compaction requested=true 2023-04-25 06:54:22,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:22,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,392 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41654e79[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:22,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:22,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:22,400 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:22,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:22,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:22,405 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:22,405 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:22,405 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:22,405 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:22,405 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:22,407 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,407 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,407 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:22,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:22,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:22,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:22,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:22,412 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:22,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:22,412 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:22,413 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:22,513 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:22,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:22,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:22,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:22,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:22,516 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:22,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:22,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:22,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:22,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:22,518 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:22,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:22,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:22,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:22,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:22,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:22,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:22,522 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:22,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:22,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:22,524 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:22,524 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:22,524 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:54:22,524 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:22,524 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:22,525 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:22,525 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:22,525 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:22,525 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:22,555 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c122df2ffbc84038b40cd92cc787f53f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c122df2ffbc84038b40cd92cc787f53f 2023-04-25 06:54:22,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:22,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405722731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405722732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405722733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405722733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405722734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405722837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405722837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405722837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405722837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:22,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405722838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:22,962 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=138, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/ecbeb98889e243e593c4a7839fdbed32 2023-04-25 06:54:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405723044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405723044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405723045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405723046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405723046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405723352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405723352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405723352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405723353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405723354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,382 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/270e71513fba4d588b93405a6bb79b9c 2023-04-25 06:54:23,807 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/baed3152b7e24cf48260899f7546988b 2023-04-25 06:54:23,811 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/ecbeb98889e243e593c4a7839fdbed32 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ecbeb98889e243e593c4a7839fdbed32 2023-04-25 06:54:23,815 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ecbeb98889e243e593c4a7839fdbed32, entries=150, sequenceid=138, filesize=25.3 K 2023-04-25 06:54:23,816 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/270e71513fba4d588b93405a6bb79b9c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/270e71513fba4d588b93405a6bb79b9c 2023-04-25 06:54:23,821 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/270e71513fba4d588b93405a6bb79b9c, entries=150, sequenceid=138, filesize=11.8 K 2023-04-25 06:54:23,822 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/baed3152b7e24cf48260899f7546988b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/baed3152b7e24cf48260899f7546988b 2023-04-25 06:54:23,827 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/baed3152b7e24cf48260899f7546988b, entries=150, sequenceid=138, filesize=11.8 K 2023-04-25 06:54:23,828 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for de84d6c21b0ed9e7c77166f8a0775f87 in 1303ms, sequenceid=138, compaction requested=true 2023-04-25 06:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,830 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d9b69e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:23,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:23,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:23,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:23,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:23,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:23,835 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:23,839 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:23,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:23,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:23,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:23,839 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:23,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:23,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:23,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:23,843 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,844 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:23,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:23,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:23,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:23,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:23,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:23,852 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:23,852 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:23,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:23,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:23,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:54:23,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:23,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:23,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:23,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:23,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:23,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:23,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405723873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405723874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405723874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405723874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405723875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,890 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425de8ce85419f244f1b98e69b8ca3504a0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425de8ce85419f244f1b98e69b8ca3504a0 2023-04-25 06:54:23,952 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:23,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:23,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:23,957 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:23,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:23,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:23,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:23,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:23,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:23,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:23,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:23,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:23,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:23,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:23,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:23,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:23,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:23,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:23,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:23,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,968 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ef7b0c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:23,970 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:23,970 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:23,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:23,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405723977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:23,979 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:23,982 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:23,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:23,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:23,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:23,983 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405723978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405723979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405723979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:23,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405723979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,985 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,985 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:23,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,985 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:23,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:23,987 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,995 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:23,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:23,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:23,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:23,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:23,995 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:23,995 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:23,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:23,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,096 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:24,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:24,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:24,099 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:24,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:24,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:24,100 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,100 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,101 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:24,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:24,103 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:24,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:24,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,105 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f8349c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:24,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:24,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:24,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:24,111 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:24,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:24,114 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:24,114 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:24,114 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:24,114 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:24,114 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:24,115 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,115 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,115 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,116 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,117 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:24,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:24,120 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:24,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,120 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:24,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405724180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405724184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405724184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405724184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405724185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,221 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:24,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:24,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:24,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:24,234 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:24,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:24,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:24,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,236 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:24,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:24,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:24,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:24,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:24,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:24,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:24,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:24,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:24,242 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:24,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@885a81c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:24,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:24,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,245 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:24,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:24,247 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:24,250 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:24,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:24,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:24,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:24,250 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:24,253 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,253 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,253 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,253 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,258 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,258 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,258 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,258 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,258 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:24,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:24,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:24,258 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,259 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,259 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:24,259 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,259 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:24,259 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,259 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,298 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/f8930b3cd80a44968978a95794700f4f 2023-04-25 06:54:24,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/e4ae318ca0444bf6ba6ecb58cbf8292f 2023-04-25 06:54:24,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/077bd8cbc86348a2b9f524b4f9e96624 2023-04-25 06:54:24,329 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/f8930b3cd80a44968978a95794700f4f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f8930b3cd80a44968978a95794700f4f 2023-04-25 06:54:24,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f8930b3cd80a44968978a95794700f4f, entries=300, sequenceid=157, filesize=45.8 K 2023-04-25 06:54:24,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/e4ae318ca0444bf6ba6ecb58cbf8292f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e4ae318ca0444bf6ba6ecb58cbf8292f 2023-04-25 06:54:24,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e4ae318ca0444bf6ba6ecb58cbf8292f, entries=150, sequenceid=157, filesize=11.8 K 2023-04-25 06:54:24,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/077bd8cbc86348a2b9f524b4f9e96624 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/077bd8cbc86348a2b9f524b4f9e96624 2023-04-25 06:54:24,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/077bd8cbc86348a2b9f524b4f9e96624, entries=150, sequenceid=157, filesize=11.8 K 2023-04-25 06:54:24,344 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for de84d6c21b0ed9e7c77166f8a0775f87 in 486ms, sequenceid=157, compaction requested=true 2023-04-25 06:54:24,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:24,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:24,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:24,344 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:24,344 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:24,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:24,346 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 150176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:24,346 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59902 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:24,346 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/A is initiating minor compaction (all files) 2023-04-25 06:54:24,346 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/B is initiating minor compaction (all files) 2023-04-25 06:54:24,346 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/A in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,346 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/B in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,346 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5df193c518054c16a09e9f5492eb9c36, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f09578c3d7dd43948975526254778a08, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/69406afb6e5d47fc8526170ae2c98670, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ecbeb98889e243e593c4a7839fdbed32, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f8930b3cd80a44968978a95794700f4f] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=146.7 K 2023-04-25 06:54:24,346 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/dfdb2b8600eb4872ac614a00f29c8b7b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/322184d7ed4f42f7921f321229e526aa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b64deabbd0b643d8a0a963b553676b9f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/270e71513fba4d588b93405a6bb79b9c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e4ae318ca0444bf6ba6ecb58cbf8292f] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=58.5 K 2023-04-25 06:54:24,347 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5df193c518054c16a09e9f5492eb9c36, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405659669 2023-04-25 06:54:24,347 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting dfdb2b8600eb4872ac614a00f29c8b7b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405659669 2023-04-25 06:54:24,347 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting f09578c3d7dd43948975526254778a08, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682405659803 2023-04-25 06:54:24,347 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 322184d7ed4f42f7921f321229e526aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682405659803 2023-04-25 06:54:24,348 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 69406afb6e5d47fc8526170ae2c98670, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682405660946 2023-04-25 06:54:24,348 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b64deabbd0b643d8a0a963b553676b9f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682405660946 2023-04-25 06:54:24,348 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting ecbeb98889e243e593c4a7839fdbed32, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1682405661592 2023-04-25 06:54:24,348 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 270e71513fba4d588b93405a6bb79b9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1682405661592 2023-04-25 06:54:24,348 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting f8930b3cd80a44968978a95794700f4f, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405662730 2023-04-25 06:54:24,349 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e4ae318ca0444bf6ba6ecb58cbf8292f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405662730 2023-04-25 06:54:24,359 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:24,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:24,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:24,375 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#B#compaction#139 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:24,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:24,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:24,380 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:24,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:24,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:24,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,382 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,383 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,389 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:24,393 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#A#compaction#138 average throughput is 0.68 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:24,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:24,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:24,408 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/67a0389ecd74424eab4716016cbdf175 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/67a0389ecd74424eab4716016cbdf175 2023-04-25 06:54:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:24,410 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,410 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:24,410 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:54:24,410 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:24,410 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:24,411 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:24,411 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:24,411 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:24,411 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:24,420 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/B of de84d6c21b0ed9e7c77166f8a0775f87 into 67a0389ecd74424eab4716016cbdf175(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:24,421 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:24,421 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/B, priority=11, startTime=1682405664344; duration=0sec 2023-04-25 06:54:24,421 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:24,421 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:24,424 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59902 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:24,424 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/C is initiating minor compaction (all files) 2023-04-25 06:54:24,424 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/C in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,424 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/93249af5659643c98fc52d4396dbc9dc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fc2f49cbd0ee43d7bee804672b03051f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9b3d972d2d44af28a82452447c6fbbc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/baed3152b7e24cf48260899f7546988b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/077bd8cbc86348a2b9f524b4f9e96624] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=58.5 K 2023-04-25 06:54:24,425 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 93249af5659643c98fc52d4396dbc9dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405659669 2023-04-25 06:54:24,426 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting fc2f49cbd0ee43d7bee804672b03051f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682405659803 2023-04-25 06:54:24,428 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a9b3d972d2d44af28a82452447c6fbbc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682405660946 2023-04-25 06:54:24,428 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting baed3152b7e24cf48260899f7546988b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1682405661592 2023-04-25 06:54:24,429 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 077bd8cbc86348a2b9f524b4f9e96624, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405662730 2023-04-25 06:54:24,445 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#C#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:24,450 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251432c143c55d4bcfbaddc79c7ece09f0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251432c143c55d4bcfbaddc79c7ece09f0 2023-04-25 06:54:24,457 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/c0631597a23040c3b6eb2a67dedc06be 2023-04-25 06:54:24,474 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/5c5752d188ff48d698ea1b512deb3844 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/5c5752d188ff48d698ea1b512deb3844 2023-04-25 06:54:24,480 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/C of de84d6c21b0ed9e7c77166f8a0775f87 into 5c5752d188ff48d698ea1b512deb3844(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:24,480 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:24,481 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/C, priority=11, startTime=1682405664344; duration=0sec 2023-04-25 06:54:24,481 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:24,483 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/fec28c6f7ae54633a84fa21419bbf428 2023-04-25 06:54:24,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:24,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:24,505 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/9aea241502c44fe7927c4b50a340c4f0 2023-04-25 06:54:24,510 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/c0631597a23040c3b6eb2a67dedc06be as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/c0631597a23040c3b6eb2a67dedc06be 2023-04-25 06:54:24,514 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/c0631597a23040c3b6eb2a67dedc06be, entries=150, sequenceid=170, filesize=25.3 K 2023-04-25 06:54:24,515 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/fec28c6f7ae54633a84fa21419bbf428 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fec28c6f7ae54633a84fa21419bbf428 2023-04-25 06:54:24,519 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fec28c6f7ae54633a84fa21419bbf428, entries=150, sequenceid=170, filesize=11.8 K 2023-04-25 06:54:24,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/9aea241502c44fe7927c4b50a340c4f0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/9aea241502c44fe7927c4b50a340c4f0 2023-04-25 06:54:24,525 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/9aea241502c44fe7927c4b50a340c4f0, entries=150, sequenceid=170, filesize=11.8 K 2023-04-25 06:54:24,526 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for de84d6c21b0ed9e7c77166f8a0775f87 in 116ms, sequenceid=170, compaction requested=false 2023-04-25 06:54:24,527 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:24,527 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:24,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:24,531 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 06:54:24,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:24,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:24,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:24,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:24,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:24,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:24,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:24,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d3d82d4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:24,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:24,538 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:24,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:24,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:24,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:24,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:24,551 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:24,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:24,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:24,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:24,551 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:24,556 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,556 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,556 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:24,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:24,563 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:24,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,564 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,564 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,564 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:24,564 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:24,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405664550, max timestamp=1682405664566Num uniques 859; 2023-04-25 06:54:24,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:54:24,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405664550, max timestamp=1682405664566Num uniques 859; 2023-04-25 06:54:24,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:54:24,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405664550, max timestamp=1682405664566Num uniques 859; 2023-04-25 06:54:24,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:54:24,588 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255554a0fffeb34458b7cf04af86280028 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255554a0fffeb34458b7cf04af86280028 2023-04-25 06:54:24,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405724585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405724586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405724587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405724588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405724592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,664 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:24,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:24,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:24,667 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:24,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:24,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:24,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,671 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,671 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,671 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:24,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:24,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:24,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:24,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:24,673 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,673 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:24,673 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:24,673 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:24,674 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3265b564[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:24,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:24,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,678 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:24,678 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,678 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:24,679 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:24,682 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:24,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:24,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:24,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:24,682 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:24,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:24,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:24,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:24,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,689 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:24,689 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:24,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405724693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405724693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405724694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405724694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405724697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,789 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:24,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:24,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:24,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:24,792 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:24,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:24,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:24,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:24,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:24,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:24,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:24,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:24,797 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:24,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@109b2b60[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:24,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:24,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,802 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:24,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:24,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:24,807 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:24,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:24,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:24,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:24,807 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:24,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,811 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,817 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,817 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,817 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,817 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,817 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:24,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:24,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,818 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:24,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,818 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:24,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405724896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405724896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405724898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405724899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:24,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405724899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,918 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:24,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:24,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:24,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:24,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:24,921 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:24,921 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,921 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:24,921 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:24,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,924 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,924 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,924 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:24,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:24,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:24,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:24,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:24,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:24,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:24,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:24,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:24,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:24,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357ba427[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:24,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:24,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,930 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:24,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:24,932 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:24,934 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:24,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:24,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:24,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:24,935 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:24,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:24,940 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:24,941 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:24,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:24,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:24,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:24,944 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:24,944 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:24,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:24,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:24,998 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/34ceeb47824a48ffaf635fec2e50f992 2023-04-25 06:54:25,044 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,047 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,047 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,047 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,047 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,048 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,048 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,049 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,050 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,050 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,050 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,050 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:25,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,052 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@461ab7e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:25,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:25,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,055 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:25,056 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,056 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:25,058 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:25,061 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:25,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:25,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:25,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:25,061 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:25,064 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,064 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,064 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,065 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:25,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,069 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:25,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,069 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:25,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,169 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,173 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,177 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,177 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,177 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:25,177 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:25,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b30f003[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:25,182 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:25,182 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,183 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:25,183 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,183 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:25,184 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:25,186 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:25,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:25,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:25,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:25,186 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:25,188 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,188 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,188 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,189 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,190 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:25,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:25,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,197 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:25,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:25,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,197 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:25,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405725201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405725202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405725203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405725204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405725205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,249 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/a5d9d244e57d4cf8895ae6a623023b7c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a5d9d244e57d4cf8895ae6a623023b7c 2023-04-25 06:54:25,255 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/A of de84d6c21b0ed9e7c77166f8a0775f87 into a5d9d244e57d4cf8895ae6a623023b7c(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:25,255 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:25,255 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/A, priority=11, startTime=1682405664344; duration=0sec 2023-04-25 06:54:25,255 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:25,297 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,301 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,302 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,302 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,303 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,303 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:25,306 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5239082e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:25,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:25,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,309 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:25,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:25,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:25,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:25,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:25,314 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:25,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:25,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:25,315 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:25,315 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:25,316 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,316 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,316 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,318 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,319 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:25,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:25,324 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,324 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:25,324 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:25,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/f27ea0cf967846d58a1faad589debf12 2023-04-25 06:54:25,425 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,428 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,431 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,431 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,433 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,433 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,434 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:25,434 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,434 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:25,434 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:25,434 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,434 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:25,434 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:25,436 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,436 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,436 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,436 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a8bc4d2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:25,438 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:25,439 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,440 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:25,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:25,440 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,440 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:25,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:25,442 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,442 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:25,442 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,442 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:25,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:25,446 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:25,446 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:25,446 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:25,446 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:25,446 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:25,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,448 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:25,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:25,452 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,452 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:25,452 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:25,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,553 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,556 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,558 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:25,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:25,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b8b2a6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:25,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:25,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,567 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:25,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:25,567 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,567 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:25,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:25,568 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:25,569 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,568 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,569 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:25,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:25,571 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:25,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:25,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:25,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:25,571 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:25,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,573 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:25,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,578 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:25,579 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:25,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,679 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,684 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:25,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:25,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:25,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:25,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:25,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,691 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,691 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,691 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:25,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f14fc3a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:25,693 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:25,693 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,694 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,694 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:25,694 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:25,694 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:25,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,696 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:25,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:25,698 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:25,698 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:25,698 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:25,698 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:25,698 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:25,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405725702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,705 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405725706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405725709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405725710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:25,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405725710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,717 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,717 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,717 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,717 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,717 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,718 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,717 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:25,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:25,718 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,718 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,718 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,718 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:25,718 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:25,818 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/0bea725a0f4c49dc801ebce40e17f3cb 2023-04-25 06:54:25,823 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,826 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,826 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,826 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,828 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,828 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,828 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:25,828 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:25,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/34ceeb47824a48ffaf635fec2e50f992 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/34ceeb47824a48ffaf635fec2e50f992 2023-04-25 06:54:25,831 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f6e98e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:25,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:25,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,836 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:25,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:25,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/34ceeb47824a48ffaf635fec2e50f992, entries=250, sequenceid=183, filesize=38.9 K 2023-04-25 06:54:25,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/f27ea0cf967846d58a1faad589debf12 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f27ea0cf967846d58a1faad589debf12 2023-04-25 06:54:25,838 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f27ea0cf967846d58a1faad589debf12, entries=150, sequenceid=183, filesize=11.8 K 2023-04-25 06:54:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:25,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:25,846 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:25,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:25,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:25,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/0bea725a0f4c49dc801ebce40e17f3cb as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/0bea725a0f4c49dc801ebce40e17f3cb 2023-04-25 06:54:25,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:25,846 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:25,848 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:25,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,850 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,852 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,852 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:25,854 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:25,854 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:25,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:25,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:25,855 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:25,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:25,855 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:25,855 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:25,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:25,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/0bea725a0f4c49dc801ebce40e17f3cb, entries=150, sequenceid=183, filesize=11.8 K 2023-04-25 06:54:25,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for de84d6c21b0ed9e7c77166f8a0775f87 in 1327ms, sequenceid=183, compaction requested=true 2023-04-25 06:54:25,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:25,859 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:25,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:25,859 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:25,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:25,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:25,860 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:25,860 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:25,861 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/A is initiating minor compaction (all files) 2023-04-25 06:54:25,861 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/B is initiating minor compaction (all files) 2023-04-25 06:54:25,861 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/A in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,861 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/B in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,861 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a5d9d244e57d4cf8895ae6a623023b7c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/c0631597a23040c3b6eb2a67dedc06be, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/34ceeb47824a48ffaf635fec2e50f992] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=89.9 K 2023-04-25 06:54:25,861 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/67a0389ecd74424eab4716016cbdf175, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fec28c6f7ae54633a84fa21419bbf428, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f27ea0cf967846d58a1faad589debf12] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=35.6 K 2023-04-25 06:54:25,861 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a5d9d244e57d4cf8895ae6a623023b7c, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405662730 2023-04-25 06:54:25,861 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 67a0389ecd74424eab4716016cbdf175, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405662730 2023-04-25 06:54:25,862 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting fec28c6f7ae54633a84fa21419bbf428, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1682405663871 2023-04-25 06:54:25,862 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c0631597a23040c3b6eb2a67dedc06be, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1682405663871 2023-04-25 06:54:25,862 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting f27ea0cf967846d58a1faad589debf12, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682405664505 2023-04-25 06:54:25,862 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 34ceeb47824a48ffaf635fec2e50f992, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682405664489 2023-04-25 06:54:25,870 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#B#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:25,885 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#A#compaction#148 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:25,892 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/fab318e413334aae82befb9123f23a45 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fab318e413334aae82befb9123f23a45 2023-04-25 06:54:25,900 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/B of de84d6c21b0ed9e7c77166f8a0775f87 into fab318e413334aae82befb9123f23a45(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:25,900 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:25,900 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/B, priority=13, startTime=1682405665859; duration=0sec 2023-04-25 06:54:25,900 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:25,900 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:25,902 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:25,902 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/C is initiating minor compaction (all files) 2023-04-25 06:54:25,902 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/C in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,902 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/5c5752d188ff48d698ea1b512deb3844, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/9aea241502c44fe7927c4b50a340c4f0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/0bea725a0f4c49dc801ebce40e17f3cb] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=35.6 K 2023-04-25 06:54:25,903 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5c5752d188ff48d698ea1b512deb3844, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682405662730 2023-04-25 06:54:25,903 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9aea241502c44fe7927c4b50a340c4f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1682405663871 2023-04-25 06:54:25,903 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 0bea725a0f4c49dc801ebce40e17f3cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682405664505 2023-04-25 06:54:25,913 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#C#compaction#149 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:25,913 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/ffd8446dfade43db9bb50256a58bee6d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ffd8446dfade43db9bb50256a58bee6d 2023-04-25 06:54:25,919 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/A of de84d6c21b0ed9e7c77166f8a0775f87 into ffd8446dfade43db9bb50256a58bee6d(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:25,920 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:25,920 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/A, priority=13, startTime=1682405665859; duration=0sec 2023-04-25 06:54:25,920 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:25,955 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:25,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:25,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:25,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:25,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:25,959 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:25,959 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,959 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:25,959 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:25,960 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,960 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:25,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:25,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:25,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:25,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:25,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:25,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:25,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:25,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:25,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:25,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:25,963 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:25,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:26,327 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/19ef959183584f059961b9efec612647 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/19ef959183584f059961b9efec612647 2023-04-25 06:54:26,332 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/C of de84d6c21b0ed9e7c77166f8a0775f87 into 19ef959183584f059961b9efec612647(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:26,333 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:26,333 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/C, priority=13, startTime=1682405665859; duration=0sec 2023-04-25 06:54:26,333 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:26,382 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042549729fad9ab140ebb350fcca518036d0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042549729fad9ab140ebb350fcca518036d0 2023-04-25 06:54:26,404 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/a34e469f55a142588bd5f0193cbc6f99 2023-04-25 06:54:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:26,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:26,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405726716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405726717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405726718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405726719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405726719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,818 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/29a279d207ff46b8855cb5be417044c6 2023-04-25 06:54:26,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405726820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405726821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405726821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405726822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,830 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/1a3962cbb3694c008662423497f0fc74 2023-04-25 06:54:26,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:26,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405726831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,835 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/a34e469f55a142588bd5f0193cbc6f99 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a34e469f55a142588bd5f0193cbc6f99 2023-04-25 06:54:26,839 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a34e469f55a142588bd5f0193cbc6f99, entries=150, sequenceid=220, filesize=25.3 K 2023-04-25 06:54:26,841 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/29a279d207ff46b8855cb5be417044c6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/29a279d207ff46b8855cb5be417044c6 2023-04-25 06:54:26,846 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/29a279d207ff46b8855cb5be417044c6, entries=150, sequenceid=220, filesize=11.8 K 2023-04-25 06:54:26,847 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/1a3962cbb3694c008662423497f0fc74 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1a3962cbb3694c008662423497f0fc74 2023-04-25 06:54:26,852 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1a3962cbb3694c008662423497f0fc74, entries=150, sequenceid=220, filesize=11.8 K 2023-04-25 06:54:26,853 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for de84d6c21b0ed9e7c77166f8a0775f87 in 890ms, sequenceid=220, compaction requested=false 2023-04-25 06:54:26,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:26,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:26,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:26,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51bf16b6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:26,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:26,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,858 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:26,858 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,858 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:26,863 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:26,864 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:26,864 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:26,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:26,866 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:26,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:26,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:26,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:26,866 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:26,868 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,868 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,868 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:26,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,871 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,877 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:26,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:26,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:26,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:26,878 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,878 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:26,878 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:26,878 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:26,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:26,978 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:26,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:26,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:26,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:26,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:26,982 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:26,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:26,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:26,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:26,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:26,984 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,984 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:26,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:26,985 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:26,985 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:26,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:26,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:26,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:26,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:26,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:26,986 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 06:54:26,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:26,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:26,987 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:26,987 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:26,987 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:26,987 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:27,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:27,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:27,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405727039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405727040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405727041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405667037, max timestamp=1682405667041Num uniques 859; 2023-04-25 06:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:27,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682405667032, max timestamp=1682405667041Num uniques 859; 2023-04-25 06:54:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 06:54:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405667037, max timestamp=1682405667041Num uniques 859; 2023-04-25 06:54:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:27,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405727072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405727074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405727144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405727144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405727144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405727175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405727176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405727347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405727347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405727348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405727381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51360 deadline: 1682405727382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,404 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042564591901391644359ff5b0c5b8cbcedf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042564591901391644359ff5b0c5b8cbcedf 2023-04-25 06:54:27,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405727649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405727650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405727651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405727685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:27,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405727685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:27,810 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/a4698afa737441e99eb8e636c9161f2a 2023-04-25 06:54:28,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51374 deadline: 1682405728153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51406 deadline: 1682405728153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:28,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405728155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:28,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405728189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:28,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405728190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,229 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/b3bfe5ad6f55401cb93b3e094c231c2b 2023-04-25 06:54:28,641 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/3ece4f001f9540ff9b4f05c21526d4d2 2023-04-25 06:54:28,648 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/a4698afa737441e99eb8e636c9161f2a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a4698afa737441e99eb8e636c9161f2a 2023-04-25 06:54:28,652 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a4698afa737441e99eb8e636c9161f2a, entries=150, sequenceid=232, filesize=25.3 K 2023-04-25 06:54:28,654 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/b3bfe5ad6f55401cb93b3e094c231c2b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b3bfe5ad6f55401cb93b3e094c231c2b 2023-04-25 06:54:28,665 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b3bfe5ad6f55401cb93b3e094c231c2b, entries=150, sequenceid=232, filesize=11.8 K 2023-04-25 06:54:28,666 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/3ece4f001f9540ff9b4f05c21526d4d2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/3ece4f001f9540ff9b4f05c21526d4d2 2023-04-25 06:54:28,671 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/3ece4f001f9540ff9b4f05c21526d4d2, entries=150, sequenceid=232, filesize=11.8 K 2023-04-25 06:54:28,673 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for de84d6c21b0ed9e7c77166f8a0775f87 in 1687ms, sequenceid=232, compaction requested=true 2023-04-25 06:54:28,673 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:28,673 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:28,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,677 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:28,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27e4a3a5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:28,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:28,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,680 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:28,681 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,681 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:28,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:28,685 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:28,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:28,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:28,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:28,685 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:28,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:28,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:28,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:28,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:28,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:28,696 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:28,696 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,696 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,696 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:28,696 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,696 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:28,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:28,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:28,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:28,696 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:28,696 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:28,796 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:28,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:28,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:28,801 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:28,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:28,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:28,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:28,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:28,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:28,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:28,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:28,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:28,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:28,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:28,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:28,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:28,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:28,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:28,806 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=131.94 KB heapSize=370.53 KB 2023-04-25 06:54:28,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:28,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:28,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:28,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:28,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:28,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:28,836 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425a7b4763ef9e64862a7c7412b6661368a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a7b4763ef9e64862a7c7412b6661368a 2023-04-25 06:54:29,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:29,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:29,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405729171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405729176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51398 deadline: 1682405729177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405729194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51390 deadline: 1682405729197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,250 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=267, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/498f936d99654c98bbb10e28dc55b098 2023-04-25 06:54:29,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405729278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405729281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405729281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405729481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405729484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405729485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,664 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c 2023-04-25 06:54:29,679 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/2a32678bd5a949ab8f912c53de15b101 2023-04-25 06:54:29,684 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/498f936d99654c98bbb10e28dc55b098 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/498f936d99654c98bbb10e28dc55b098 2023-04-25 06:54:29,688 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/498f936d99654c98bbb10e28dc55b098, entries=150, sequenceid=267, filesize=25.4 K 2023-04-25 06:54:29,689 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c 2023-04-25 06:54:29,693 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c, entries=150, sequenceid=267, filesize=11.9 K 2023-04-25 06:54:29,694 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/2a32678bd5a949ab8f912c53de15b101 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/2a32678bd5a949ab8f912c53de15b101 2023-04-25 06:54:29,699 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/2a32678bd5a949ab8f912c53de15b101, entries=150, sequenceid=267, filesize=11.9 K 2023-04-25 06:54:29,700 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~370.48 KB/379376, currentSize=53.67 KB/54960 for de84d6c21b0ed9e7c77166f8a0775f87 in 894ms, sequenceid=267, compaction requested=true 2023-04-25 06:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:29,704 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f196e4f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:29,706 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:29,706 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,708 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:29,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:29,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:29,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:29,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:29,712 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:29,712 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:29,712 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:29,712 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:29,712 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:29,713 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,713 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,717 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:29,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:29,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:29,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:29,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:29,719 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:29,720 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:29,720 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,720 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,720 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,720 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:29,720 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:29,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:29,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:54:29,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:29,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:29,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:29,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:29,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:29,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:29,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405729806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405729809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405729809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,820 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:29,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:29,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:29,824 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:29,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:29,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:29,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:29,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:29,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:29,827 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,828 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,828 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:29,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:29,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:29,829 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:29,829 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:29,829 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:29,829 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:29,831 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:29,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d2158bb[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:29,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:29,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:29,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:29,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:29,835 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,835 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,835 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:29,835 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:29,838 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:29,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:29,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:29,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:29,838 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:29,840 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,840 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,840 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:29,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,841 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:29,844 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:29,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:29,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,844 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:29,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:29,844 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:29,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:29,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405729910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405729911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:29,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405729913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,945 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:29,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:29,948 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:29,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:29,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:29,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:29,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:29,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:29,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:29,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:29,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:29,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:29,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:29,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:29,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:29,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:29,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:29,952 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:29,952 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:29,952 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:29,952 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:29,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:29,954 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:29,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@402f10f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:29,956 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:29,956 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,958 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,959 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,959 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:29,959 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:29,959 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:29,959 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:29,960 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:29,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:29,963 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:29,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:29,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:29,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:29,963 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:29,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:29,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:29,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,967 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:29,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:29,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:29,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:29,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:29,969 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:29,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:29,970 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:29,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:29,970 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:29,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:29,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:29,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,070 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,073 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,073 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:30,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,078 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,078 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,079 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67983a1[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,088 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,091 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,094 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,094 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,095 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,095 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,095 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:30,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:30,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,099 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,099 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:30,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:30,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405730113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405730114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405730119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,199 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,203 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,205 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,206 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,206 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,206 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c4ed974fbe4b40b78e67f438fe29ee23 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c4ed974fbe4b40b78e67f438fe29ee23 2023-04-25 06:54:30,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:30,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,208 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,208 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,208 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,208 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,209 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,210 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,210 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,210 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6276ee37[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,212 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,212 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,213 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,213 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,213 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,215 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,218 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,218 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,221 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,223 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,223 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,223 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:30,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:30,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:30,224 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:30,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,223 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,224 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:30,223 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,224 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,224 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,224 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,224 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,324 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,328 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,329 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,329 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:30,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,332 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,332 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,332 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,332 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,335 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b59642[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,337 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,338 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,338 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,338 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,340 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,342 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,342 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,344 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,344 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,344 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:30,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,354 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:30,354 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:30,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,355 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405730432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405730432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405730432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,455 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,458 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,459 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,460 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,461 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,461 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,461 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,461 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,463 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@562c8d72[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,466 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,474 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,474 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,474 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,474 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,474 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,475 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,475 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,475 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,476 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,479 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:30,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,479 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:30,479 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:30,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,580 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,583 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,586 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,586 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:30,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,587 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,587 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,587 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,592 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,592 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,592 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,592 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,593 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,593 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,596 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,596 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c02789d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,598 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,598 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,598 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,600 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,603 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,603 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,603 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,604 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,604 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,605 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,605 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,606 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,610 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:30,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:30,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,610 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:30,610 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:30,613 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/33691276e67641a7959a382b505bc178 2023-04-25 06:54:30,710 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,714 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,715 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,715 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49503a7c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,722 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,722 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,722 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,722 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,724 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,726 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,726 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,728 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,728 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,728 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,729 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,731 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,733 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:30,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:30,733 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,733 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:30,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,834 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,837 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:30,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,841 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,841 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,841 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,841 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,844 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,844 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,845 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@203ea292[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,847 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,849 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,849 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,849 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,851 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,851 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,853 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,853 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,853 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,853 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,855 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,861 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,861 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,861 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,861 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,861 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:30,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:30,861 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,862 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,862 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:30,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405730936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405730939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:30,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405730940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,962 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:30,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:30,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:30,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:30,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:30,965 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:30,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:30,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:30,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,967 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:30,967 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:30,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:30,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:30,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:30,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:30,969 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:30,969 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:30,969 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:30,970 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,970 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:30,970 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:30,970 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:30,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,972 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,972 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,972 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12cbecfa[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:30,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:30,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,975 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:30,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:30,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:30,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:30,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:30,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:30,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:30,989 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:30,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:30,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:30,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:30,989 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:30,991 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,991 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,991 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:30,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:30,992 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:31,000 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:31,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:31,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:31,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:31,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:31,004 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:31,004 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:31,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:31,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/324709ec0ca54ffeb12977c3a6b73de4 2023-04-25 06:54:31,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/786dc170579b4e4ba77ff5534e17e830 2023-04-25 06:54:31,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/33691276e67641a7959a382b505bc178 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/33691276e67641a7959a382b505bc178 2023-04-25 06:54:31,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/33691276e67641a7959a382b505bc178, entries=200, sequenceid=279, filesize=32.3 K 2023-04-25 06:54:31,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/324709ec0ca54ffeb12977c3a6b73de4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/324709ec0ca54ffeb12977c3a6b73de4 2023-04-25 06:54:31,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/324709ec0ca54ffeb12977c3a6b73de4, entries=150, sequenceid=279, filesize=11.9 K 2023-04-25 06:54:31,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/786dc170579b4e4ba77ff5534e17e830 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/786dc170579b4e4ba77ff5534e17e830 2023-04-25 06:54:31,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/786dc170579b4e4ba77ff5534e17e830, entries=150, sequenceid=279, filesize=11.9 K 2023-04-25 06:54:31,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for de84d6c21b0ed9e7c77166f8a0775f87 in 1301ms, sequenceid=279, compaction requested=true 2023-04-25 06:54:31,086 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:31,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:31,086 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:31,087 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:31,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:31,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:31,088 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137264 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:31,088 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/A is initiating minor compaction (all files) 2023-04-25 06:54:31,088 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60924 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:31,089 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/A in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:31,089 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/B is initiating minor compaction (all files) 2023-04-25 06:54:31,089 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/B in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:31,089 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ffd8446dfade43db9bb50256a58bee6d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a34e469f55a142588bd5f0193cbc6f99, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a4698afa737441e99eb8e636c9161f2a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/498f936d99654c98bbb10e28dc55b098, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/33691276e67641a7959a382b505bc178] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=134.0 K 2023-04-25 06:54:31,089 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fab318e413334aae82befb9123f23a45, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/29a279d207ff46b8855cb5be417044c6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b3bfe5ad6f55401cb93b3e094c231c2b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/324709ec0ca54ffeb12977c3a6b73de4] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=59.5 K 2023-04-25 06:54:31,089 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting ffd8446dfade43db9bb50256a58bee6d, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682405664505 2023-04-25 06:54:31,089 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting fab318e413334aae82befb9123f23a45, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682405664505 2023-04-25 06:54:31,090 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a34e469f55a142588bd5f0193cbc6f99, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682405664584 2023-04-25 06:54:31,090 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 29a279d207ff46b8855cb5be417044c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682405664584 2023-04-25 06:54:31,090 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a4698afa737441e99eb8e636c9161f2a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682405666716 2023-04-25 06:54:31,090 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b3bfe5ad6f55401cb93b3e094c231c2b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682405666716 2023-04-25 06:54:31,091 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 498f936d99654c98bbb10e28dc55b098, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682405667065 2023-04-25 06:54:31,091 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e09ebe5e84804c2ab2f8ab3d15c1aa3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682405667065 2023-04-25 06:54:31,091 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 33691276e67641a7959a382b505bc178, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682405669168 2023-04-25 06:54:31,091 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 324709ec0ca54ffeb12977c3a6b73de4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682405669168 2023-04-25 06:54:31,104 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:31,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:31,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:31,111 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#B#compaction#163 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:31,114 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#A#compaction#162 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:31,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:31,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:31,120 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:31,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:31,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:31,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:31,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:31,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:31,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,123 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:31,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:31,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:31,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:31,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:31,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:31,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:31,138 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:31,139 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:31,139 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:54:31,139 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:31,139 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:31,139 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:31,139 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:31,139 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:31,139 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:31,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:31,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:31,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405731230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405731231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405731333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405731333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,534 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/3be026220cbb400a86c1c0f6f80481ca as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/3be026220cbb400a86c1c0f6f80481ca 2023-04-25 06:54:31,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405731535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405731535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,540 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/B of de84d6c21b0ed9e7c77166f8a0775f87 into 3be026220cbb400a86c1c0f6f80481ca(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:31,540 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:31,540 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/B, priority=11, startTime=1682405671086; duration=0sec 2023-04-25 06:54:31,540 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:31,540 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:31,543 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60924 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:31,543 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/C is initiating minor compaction (all files) 2023-04-25 06:54:31,543 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/C in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:31,543 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/19ef959183584f059961b9efec612647, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1a3962cbb3694c008662423497f0fc74, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/3ece4f001f9540ff9b4f05c21526d4d2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/2a32678bd5a949ab8f912c53de15b101, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/786dc170579b4e4ba77ff5534e17e830] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=59.5 K 2023-04-25 06:54:31,544 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 19ef959183584f059961b9efec612647, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682405664505 2023-04-25 06:54:31,544 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1a3962cbb3694c008662423497f0fc74, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682405664584 2023-04-25 06:54:31,545 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3ece4f001f9540ff9b4f05c21526d4d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682405666716 2023-04-25 06:54:31,546 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2a32678bd5a949ab8f912c53de15b101, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682405667065 2023-04-25 06:54:31,546 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 786dc170579b4e4ba77ff5534e17e830, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682405669168 2023-04-25 06:54:31,567 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#C#compaction#165 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:31,585 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ce9c524d600f43d7b3ba3474f4062b3c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ce9c524d600f43d7b3ba3474f4062b3c 2023-04-25 06:54:31,605 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/cb0b1fdf444642f18b6149b87cf9fff8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/cb0b1fdf444642f18b6149b87cf9fff8 2023-04-25 06:54:31,613 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/A of de84d6c21b0ed9e7c77166f8a0775f87 into cb0b1fdf444642f18b6149b87cf9fff8(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:31,613 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:31,613 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/A, priority=11, startTime=1682405671086; duration=0sec 2023-04-25 06:54:31,613 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:31,621 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/fca9c55aa01241a1b9e8e9d8f4df4d13 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fca9c55aa01241a1b9e8e9d8f4df4d13 2023-04-25 06:54:31,626 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/C of de84d6c21b0ed9e7c77166f8a0775f87 into fca9c55aa01241a1b9e8e9d8f4df4d13(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:31,626 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:31,626 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/C, priority=11, startTime=1682405671087; duration=0sec 2023-04-25 06:54:31,627 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:31,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405731837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405731840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405731942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405731944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:31,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:31,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405731951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,018 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/5e5ec9f5ced546b38d69b7fc85091aba 2023-04-25 06:54:32,032 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/37a95e6c4dc040f3a15e06da5540ed91 2023-04-25 06:54:32,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:32,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405732342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:32,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405732343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,447 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/c3f62d9f0dad4987bf61d68a6132f05c 2023-04-25 06:54:32,452 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/5e5ec9f5ced546b38d69b7fc85091aba as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5e5ec9f5ced546b38d69b7fc85091aba 2023-04-25 06:54:32,457 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5e5ec9f5ced546b38d69b7fc85091aba, entries=150, sequenceid=299, filesize=25.4 K 2023-04-25 06:54:32,458 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/37a95e6c4dc040f3a15e06da5540ed91 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/37a95e6c4dc040f3a15e06da5540ed91 2023-04-25 06:54:32,477 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/37a95e6c4dc040f3a15e06da5540ed91, entries=150, sequenceid=299, filesize=11.9 K 2023-04-25 06:54:32,479 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/c3f62d9f0dad4987bf61d68a6132f05c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/c3f62d9f0dad4987bf61d68a6132f05c 2023-04-25 06:54:32,484 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/c3f62d9f0dad4987bf61d68a6132f05c, entries=150, sequenceid=299, filesize=11.9 K 2023-04-25 06:54:32,485 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for de84d6c21b0ed9e7c77166f8a0775f87 in 1346ms, sequenceid=299, compaction requested=false 2023-04-25 06:54:32,485 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:32,485 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:32,485 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:32,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:32,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:32,498 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:32,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:32,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:32,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d82b8ae[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:32,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:32,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,502 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:32,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:32,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,502 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:32,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:32,504 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:32,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:32,507 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:32,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:32,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:32,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:32,507 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:32,509 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,509 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,509 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:32,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,510 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:32,512 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,515 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:32,515 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:32,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:32,515 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:32,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:32,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:32,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:32,516 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:32,516 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:32,515 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:32,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:32,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:32,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:32,616 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:32,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:32,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:32,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:32,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:32,620 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:32,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:32,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:32,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:32,621 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,621 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:32,622 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,622 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:32,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:32,623 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:32,623 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:32,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:32,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:32,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:32,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:32,632 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:32,632 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:32,632 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:54:32,633 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:32,633 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:32,633 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:32,633 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:32,633 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:32,633 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:33,059 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ea5b5094b8fa434f8d44e5cb6afe6ce4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ea5b5094b8fa434f8d44e5cb6afe6ce4 2023-04-25 06:54:33,078 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/03f2877d13984b40ab732cf1c744c9da 2023-04-25 06:54:33,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:33,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:33,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405673377, max timestamp=1682405673402Num uniques 859; 2023-04-25 06:54:33,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:54:33,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405673377, max timestamp=1682405673402Num uniques 859; 2023-04-25 06:54:33,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405673377, max timestamp=1682405673402Num uniques 859; 2023-04-25 06:54:33,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:54:33,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:54:33,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405733431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405733433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,507 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/ade6a286ed4b46dfad6d7506cad6c51b 2023-04-25 06:54:33,521 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/369ba48778a14468b91be12def320216 2023-04-25 06:54:33,526 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/03f2877d13984b40ab732cf1c744c9da as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/03f2877d13984b40ab732cf1c744c9da 2023-04-25 06:54:33,530 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/03f2877d13984b40ab732cf1c744c9da, entries=100, sequenceid=314, filesize=18.6 K 2023-04-25 06:54:33,531 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/ade6a286ed4b46dfad6d7506cad6c51b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/ade6a286ed4b46dfad6d7506cad6c51b 2023-04-25 06:54:33,537 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/ade6a286ed4b46dfad6d7506cad6c51b, entries=100, sequenceid=314, filesize=9.5 K 2023-04-25 06:54:33,538 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/369ba48778a14468b91be12def320216 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/369ba48778a14468b91be12def320216 2023-04-25 06:54:33,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405733535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405733537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,544 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/369ba48778a14468b91be12def320216, entries=100, sequenceid=314, filesize=9.5 K 2023-04-25 06:54:33,547 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for de84d6c21b0ed9e7c77166f8a0775f87 in 915ms, sequenceid=314, compaction requested=true 2023-04-25 06:54:33,547 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:33,547 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:33,549 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:33,549 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:33,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,550 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:33,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:33,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5549f2b8[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:33,552 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:33,552 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,553 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:33,554 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,554 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:33,556 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:33,556 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:33,556 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,556 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:33,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:33,562 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:33,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:33,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:33,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:33,562 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:33,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:33,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:33,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:33,566 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,571 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,571 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:33,571 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:33,571 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,571 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:33,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:33,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:33,572 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:33,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:33,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:33,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:33,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:33,572 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:33,572 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:33,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:33,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:33,672 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:33,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:33,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:33,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:33,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:33,676 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:33,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:33,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:33,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:33,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:33,679 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:33,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:33,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:33,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:33,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:33,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:33,681 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:33,681 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:33,681 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-25 06:54:33,681 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:33,681 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:33,682 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:33,682 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:33,682 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:33,682 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:33,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:33,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:33,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405733756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405733757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405733858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405733862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51398 deadline: 1682405733952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,956 DEBUG [Thread-888] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:54:33,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51406 deadline: 1682405733956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,961 DEBUG [Thread-884] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:54:33,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:33,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51374 deadline: 1682405733968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:33,970 DEBUG [Thread-882] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:54:34,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:34,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405734064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:34,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405734065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,101 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042585cb66d4fc50496d82b12399a37e85d0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042585cb66d4fc50496d82b12399a37e85d0 2023-04-25 06:54:34,108 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/9371d46d1a32438682339312f57e4dfe 2023-04-25 06:54:34,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:34,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405734367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:34,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405734367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,529 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/da16b9ff3a1f4b12a58868dfbef0f22f 2023-04-25 06:54:34,550 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/4538c9667e0c4b69b458dcfb7fc83e0f 2023-04-25 06:54:34,557 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/9371d46d1a32438682339312f57e4dfe as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/9371d46d1a32438682339312f57e4dfe 2023-04-25 06:54:34,565 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/9371d46d1a32438682339312f57e4dfe, entries=150, sequenceid=347, filesize=25.4 K 2023-04-25 06:54:34,566 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/da16b9ff3a1f4b12a58868dfbef0f22f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/da16b9ff3a1f4b12a58868dfbef0f22f 2023-04-25 06:54:34,573 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/da16b9ff3a1f4b12a58868dfbef0f22f, entries=150, sequenceid=347, filesize=11.9 K 2023-04-25 06:54:34,574 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/4538c9667e0c4b69b458dcfb7fc83e0f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/4538c9667e0c4b69b458dcfb7fc83e0f 2023-04-25 06:54:34,582 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/4538c9667e0c4b69b458dcfb7fc83e0f, entries=150, sequenceid=347, filesize=11.9 K 2023-04-25 06:54:34,585 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=60.38 KB/61830 for de84d6c21b0ed9e7c77166f8a0775f87 in 904ms, sequenceid=347, compaction requested=true 2023-04-25 06:54:34,586 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:34,586 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:34,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:34,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:34,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:34,586 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:34,589 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:34,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:34,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f0b1[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:34,591 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:34,591 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,592 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,592 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:34,592 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:34,592 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:34,594 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:34,594 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:34,594 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:34,594 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,594 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:34,594 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:34,594 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:34,594 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:34,594 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:34,594 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:34,595 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:34,604 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,604 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,604 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:34,605 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:34,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:34,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:34,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:34,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:34,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:34,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:34,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:34,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:34,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,610 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:34,610 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:34,610 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:34,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:34,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:34,710 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:34,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:34,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:34,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:34,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:34,714 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:34,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:34,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:34,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:34,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,715 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:34,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:34,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:34,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:34,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:34,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:34,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:34,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:34,718 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:34,718 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:34,734 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425514267140c4d4b048223c1db8746826a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425514267140c4d4b048223c1db8746826a 2023-04-25 06:54:34,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:34,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:34,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:34,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405734899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:34,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405674874, max timestamp=1682405674899Num uniques 859; 2023-04-25 06:54:34,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:34,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405674874, max timestamp=1682405674899Num uniques 859; 2023-04-25 06:54:34,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:34,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405674874, max timestamp=1682405674899Num uniques 859; 2023-04-25 06:54:34,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:34,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:34,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405734961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:35,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405735002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:35,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405735064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,141 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/89fbbc9b375a4c618077fbc1167e9e2a 2023-04-25 06:54:35,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:35,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405735207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:35,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405735267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:35,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405735511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,552 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/8914cc5b62934c678eb260929ad8b7f5 2023-04-25 06:54:35,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:35,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405735571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,962 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/29c7d09292d2446f8e843e39dc68a6e9 2023-04-25 06:54:35,967 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/89fbbc9b375a4c618077fbc1167e9e2a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/89fbbc9b375a4c618077fbc1167e9e2a 2023-04-25 06:54:35,973 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/89fbbc9b375a4c618077fbc1167e9e2a, entries=150, sequenceid=359, filesize=25.4 K 2023-04-25 06:54:35,974 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/8914cc5b62934c678eb260929ad8b7f5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8914cc5b62934c678eb260929ad8b7f5 2023-04-25 06:54:35,978 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8914cc5b62934c678eb260929ad8b7f5, entries=150, sequenceid=359, filesize=11.9 K 2023-04-25 06:54:35,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/29c7d09292d2446f8e843e39dc68a6e9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/29c7d09292d2446f8e843e39dc68a6e9 2023-04-25 06:54:35,984 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/29c7d09292d2446f8e843e39dc68a6e9, entries=150, sequenceid=359, filesize=11.9 K 2023-04-25 06:54:35,985 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for de84d6c21b0ed9e7c77166f8a0775f87 in 1267ms, sequenceid=359, compaction requested=true 2023-04-25 06:54:35,985 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:35,985 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:35,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:35,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:35,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:35,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:35,987 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:35,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:35,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:35,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dd77071[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:35,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:35,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:35,990 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:35,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:35,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:35,991 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:35,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:35,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:35,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:35,994 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:35,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:35,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:35,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:35,994 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:35,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:35,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:35,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:35,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:35,998 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:35,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:35,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,001 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,001 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,002 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:36,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:36,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-25 06:54:36,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:36,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:36,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:36,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:36,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:36,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:36,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:36,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405736047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:36,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51390 deadline: 1682405736078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,102 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,107 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,108 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,108 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:36,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:36,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:36,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:36,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:36,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:36,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:36,112 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,112 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:36,112 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:36,112 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:36,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,114 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,114 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:36,115 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:36,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4518e6ad[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:36,115 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,117 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,118 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:36,118 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:36,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:36,118 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:36,119 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:36,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:36,121 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:36,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:36,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:36,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:36,121 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:36,123 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,123 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,123 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,127 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,127 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,127 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:36,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405736151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,228 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,231 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,237 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,237 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,237 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:36,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:36,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:36,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:36,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:36,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:36,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:36,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:36,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:36,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:36,239 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:36,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:36,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:36,239 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:36,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,241 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,241 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,241 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5e0084[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:36,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:36,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,244 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,244 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:36,244 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:36,244 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:36,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:36,245 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:36,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:36,248 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:36,248 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:36,248 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:36,248 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:36,248 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:36,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,252 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,253 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:36,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,256 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,256 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,256 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,256 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,256 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,356 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,359 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,360 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:36,360 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405736359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,361 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,361 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,362 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:36,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:36,363 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,363 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:36,363 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:36,363 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:36,365 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,365 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68dad0a6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:36,367 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:36,367 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,369 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:36,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:36,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:36,369 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,369 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:36,370 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,370 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,370 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:36,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:36,373 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:36,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:36,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:36,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:36,373 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:36,375 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,375 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,375 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,375 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,377 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,379 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,380 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:36,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,436 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425b21bfe4debe94ec39c88131dc509c63b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b21bfe4debe94ec39c88131dc509c63b 2023-04-25 06:54:36,480 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,488 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:36,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:36,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:36,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:36,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:36,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:36,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:36,492 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,493 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:36,493 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:36,493 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:36,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:36,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:36,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:36,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:36,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef86134[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:36,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:36,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,497 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:36,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:36,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:36,497 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,497 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:36,498 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,498 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:36,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:36,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:36,501 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:36,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:36,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:36,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:36,501 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:36,503 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,503 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,503 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:36,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,507 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,508 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,608 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,611 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,613 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:36,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:36,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:36,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:36,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:36,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:36,617 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c57d1c8[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:36,618 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:36,619 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,626 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:36,626 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,626 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,628 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:36,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:36,631 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:36,631 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:36,631 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:36,631 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:36,631 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:36,632 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,633 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,635 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,637 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,638 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,638 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,637 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,638 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,638 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:36,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51360 deadline: 1682405736661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,738 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,741 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,741 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,741 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,741 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,744 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:36,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:36,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:36,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:36,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:36,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:36,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:36,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:36,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:36,747 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:36,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3322a8e6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:36,750 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:36,750 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,751 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:36,751 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,751 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,752 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:36,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:36,755 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:36,755 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:36,755 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:36,755 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:36,755 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:36,757 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,757 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,757 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,758 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,759 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,761 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,761 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,761 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,761 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:36,846 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/fbe322eccc7d498c8d46ab7c26041fb0 2023-04-25 06:54:36,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/74256f362b1c4f25af96e8299ee154c8 2023-04-25 06:54:36,861 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,865 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,868 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,868 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:36,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:36,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:36,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:36,870 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,870 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:36,870 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:36,870 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:36,870 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:36,872 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,873 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,873 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/1233d48411924ea5a0ba8cb2f3901bd3 2023-04-25 06:54:36,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:36,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d4311c4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:36,875 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:36,875 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,877 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,877 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:36,877 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:36,878 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,878 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:36,878 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:36,879 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:36,879 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,879 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:36,879 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/fbe322eccc7d498c8d46ab7c26041fb0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/fbe322eccc7d498c8d46ab7c26041fb0 2023-04-25 06:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:36,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:36,882 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:36,882 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:36,882 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:36,882 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:36,882 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:36,884 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,884 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/fbe322eccc7d498c8d46ab7c26041fb0, entries=200, sequenceid=395, filesize=32.3 K 2023-04-25 06:54:36,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/74256f362b1c4f25af96e8299ee154c8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/74256f362b1c4f25af96e8299ee154c8 2023-04-25 06:54:36,888 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:36,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:36,888 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,890 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/74256f362b1c4f25af96e8299ee154c8, entries=150, sequenceid=395, filesize=11.9 K 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:36,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:36,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:36,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/1233d48411924ea5a0ba8cb2f3901bd3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1233d48411924ea5a0ba8cb2f3901bd3 2023-04-25 06:54:36,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:36,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:36,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:36,893 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:36,893 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:36,893 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:36,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1233d48411924ea5a0ba8cb2f3901bd3, entries=150, sequenceid=395, filesize=11.9 K 2023-04-25 06:54:36,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for de84d6c21b0ed9e7c77166f8a0775f87 in 879ms, sequenceid=395, compaction requested=true 2023-04-25 06:54:36,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:36,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:36,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:36,898 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:54:36,898 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:54:36,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:36,900 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71286 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:54:36,900 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156882 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:54:36,900 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/B is initiating minor compaction (all files) 2023-04-25 06:54:36,900 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/A is initiating minor compaction (all files) 2023-04-25 06:54:36,900 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/B in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,900 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/A in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,900 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/3be026220cbb400a86c1c0f6f80481ca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/37a95e6c4dc040f3a15e06da5540ed91, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/ade6a286ed4b46dfad6d7506cad6c51b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/da16b9ff3a1f4b12a58868dfbef0f22f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8914cc5b62934c678eb260929ad8b7f5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/74256f362b1c4f25af96e8299ee154c8] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=69.6 K 2023-04-25 06:54:36,900 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/cb0b1fdf444642f18b6149b87cf9fff8, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5e5ec9f5ced546b38d69b7fc85091aba, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/03f2877d13984b40ab732cf1c744c9da, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/9371d46d1a32438682339312f57e4dfe, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/89fbbc9b375a4c618077fbc1167e9e2a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/fbe322eccc7d498c8d46ab7c26041fb0] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=153.2 K 2023-04-25 06:54:36,901 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3be026220cbb400a86c1c0f6f80481ca, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682405669168 2023-04-25 06:54:36,901 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting cb0b1fdf444642f18b6149b87cf9fff8, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682405669168 2023-04-25 06:54:36,901 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 37a95e6c4dc040f3a15e06da5540ed91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682405669804 2023-04-25 06:54:36,901 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5e5ec9f5ced546b38d69b7fc85091aba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682405669804 2023-04-25 06:54:36,901 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ade6a286ed4b46dfad6d7506cad6c51b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405671221 2023-04-25 06:54:36,902 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 03f2877d13984b40ab732cf1c744c9da, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405671221 2023-04-25 06:54:36,902 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting da16b9ff3a1f4b12a58868dfbef0f22f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682405673424 2023-04-25 06:54:36,902 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9371d46d1a32438682339312f57e4dfe, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682405673424 2023-04-25 06:54:36,902 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 8914cc5b62934c678eb260929ad8b7f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682405673753 2023-04-25 06:54:36,902 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 89fbbc9b375a4c618077fbc1167e9e2a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682405673753 2023-04-25 06:54:36,903 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 74256f362b1c4f25af96e8299ee154c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682405674924 2023-04-25 06:54:36,903 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting fbe322eccc7d498c8d46ab7c26041fb0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682405674924 2023-04-25 06:54:36,913 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#B#compaction#180 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:36,926 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#A#compaction#181 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:36,934 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/f0ca6c56d3224b1aa216a827c898590b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f0ca6c56d3224b1aa216a827c898590b 2023-04-25 06:54:36,946 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/B of de84d6c21b0ed9e7c77166f8a0775f87 into f0ca6c56d3224b1aa216a827c898590b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:36,946 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:36,946 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/B, priority=10, startTime=1682405676898; duration=0sec 2023-04-25 06:54:36,947 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:36,947 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:54:36,949 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71286 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:54:36,949 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): de84d6c21b0ed9e7c77166f8a0775f87/C is initiating minor compaction (all files) 2023-04-25 06:54:36,949 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of de84d6c21b0ed9e7c77166f8a0775f87/C in TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:36,949 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fca9c55aa01241a1b9e8e9d8f4df4d13, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/c3f62d9f0dad4987bf61d68a6132f05c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/369ba48778a14468b91be12def320216, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/4538c9667e0c4b69b458dcfb7fc83e0f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/29c7d09292d2446f8e843e39dc68a6e9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1233d48411924ea5a0ba8cb2f3901bd3] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp, totalSize=69.6 K 2023-04-25 06:54:36,950 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting fca9c55aa01241a1b9e8e9d8f4df4d13, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682405669168 2023-04-25 06:54:36,950 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c3f62d9f0dad4987bf61d68a6132f05c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682405669804 2023-04-25 06:54:36,950 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 369ba48778a14468b91be12def320216, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405671221 2023-04-25 06:54:36,950 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4538c9667e0c4b69b458dcfb7fc83e0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682405673424 2023-04-25 06:54:36,951 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 29c7d09292d2446f8e843e39dc68a6e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682405673753 2023-04-25 06:54:36,951 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1233d48411924ea5a0ba8cb2f3901bd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682405674924 2023-04-25 06:54:36,962 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): de84d6c21b0ed9e7c77166f8a0775f87#C#compaction#182 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:36,994 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:36,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:36,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:36,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:36,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:36,997 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:36,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:36,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:36,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:36,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:36,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:37,000 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:37,000 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:37,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:37,000 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:37,000 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:37,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:37,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:37,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:37,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:37,002 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. started... 2023-04-25 06:54:37,003 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 06:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:37,022 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425cd161127a6c9403ba50a90ebd1e241b4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cd161127a6c9403ba50a90ebd1e241b4 2023-04-25 06:54:37,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:37,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. as already flushing 2023-04-25 06:54:37,178 DEBUG [Thread-886] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28fcbe97 to 127.0.0.1:60416 2023-04-25 06:54:37,178 DEBUG [Thread-886] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682405677154, max timestamp=1682405677177Num uniques 859; 2023-04-25 06:54:37,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682405677154, max timestamp=1682405677177Num uniques 859; 2023-04-25 06:54:37,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 06:54:37,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 06:54:37,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682405677154, max timestamp=1682405677177Num uniques 859; 2023-04-25 06:54:37,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 06:54:37,181 DEBUG [Thread-893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e0a3dc to 127.0.0.1:60416 2023-04-25 06:54:37,181 DEBUG [Thread-893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,181 DEBUG [Thread-890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76aef981 to 127.0.0.1:60416 2023-04-25 06:54:37,182 DEBUG [Thread-890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,183 DEBUG [Thread-899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e0b0a9f to 127.0.0.1:60416 2023-04-25 06:54:37,183 DEBUG [Thread-899] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,184 DEBUG [Thread-895] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63d375f8 to 127.0.0.1:60416 2023-04-25 06:54:37,185 DEBUG [Thread-895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,186 DEBUG [Thread-897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1345b761 to 127.0.0.1:60416 2023-04-25 06:54:37,186 DEBUG [Thread-897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,375 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/dc201d2b99f9427e9f5e741e67c3ea08 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/dc201d2b99f9427e9f5e741e67c3ea08 2023-04-25 06:54:37,379 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/C of de84d6c21b0ed9e7c77166f8a0775f87 into dc201d2b99f9427e9f5e741e67c3ea08(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:37,379 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:37,379 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/C, priority=10, startTime=1682405676898; duration=0sec 2023-04-25 06:54:37,379 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:37,437 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=407, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/d5cab30d1624441797fc26547ebb1021 2023-04-25 06:54:37,847 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/6d82821cf1e24872833a7f7b10de6ece 2023-04-25 06:54:37,974 DEBUG [Thread-888] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18496e17 to 127.0.0.1:60416 2023-04-25 06:54:37,974 DEBUG [Thread-884] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46078b1e to 127.0.0.1:60416 2023-04-25 06:54:37,974 DEBUG [Thread-888] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,974 DEBUG [Thread-884] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:37,981 DEBUG [Thread-882] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x542e33ab to 127.0.0.1:60416 2023-04-25 06:54:37,981 DEBUG [Thread-882] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:38,146 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/b608d05ecd3f4c80acc81adc4c85600a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/b608d05ecd3f4c80acc81adc4c85600a 2023-04-25 06:54:38,150 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in de84d6c21b0ed9e7c77166f8a0775f87/A of de84d6c21b0ed9e7c77166f8a0775f87 into b608d05ecd3f4c80acc81adc4c85600a(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 06:54:38,150 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:38,150 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87., storeName=de84d6c21b0ed9e7c77166f8a0775f87/A, priority=10, startTime=1682405676898; duration=1sec 2023-04-25 06:54:38,150 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:38,257 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/a9d063d1e1834ad999ded65155c677d2 2023-04-25 06:54:38,261 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/d5cab30d1624441797fc26547ebb1021 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/d5cab30d1624441797fc26547ebb1021 2023-04-25 06:54:38,264 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/d5cab30d1624441797fc26547ebb1021, entries=150, sequenceid=407, filesize=25.4 K 2023-04-25 06:54:38,265 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/6d82821cf1e24872833a7f7b10de6ece as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/6d82821cf1e24872833a7f7b10de6ece 2023-04-25 06:54:38,268 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/6d82821cf1e24872833a7f7b10de6ece, entries=150, sequenceid=407, filesize=11.9 K 2023-04-25 06:54:38,269 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/a9d063d1e1834ad999ded65155c677d2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9d063d1e1834ad999ded65155c677d2 2023-04-25 06:54:38,273 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9d063d1e1834ad999ded65155c677d2, entries=150, sequenceid=407, filesize=11.9 K 2023-04-25 06:54:38,274 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=73.80 KB/75570 for de84d6c21b0ed9e7c77166f8a0775f87 in 1271ms, sequenceid=407, compaction requested=false 2023-04-25 06:54:38,274 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:38,274 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:38,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:38,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:38,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:38,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50ef2099[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:38,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:38,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,281 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,281 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:38,281 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:38,281 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:38,281 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,281 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:38,282 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:38,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:38,284 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:38,284 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:38,284 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:38,284 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:38,284 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:38,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:38,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:38,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:38,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:38,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:38,287 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:38,288 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:38,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:38,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:38,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:38,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:38,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:38,290 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:38,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:38,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:38,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,291 DEBUG [Thread-892] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:38,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,291 DEBUG [Thread-892] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:38,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:38,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:38,391 DEBUG [Thread-892] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:38,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 4465 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 4391 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1828 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5484 rows 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1830 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5488 rows 2023-04-25 06:54:38,392 INFO [Listener at localhost/41663] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 06:54:38,393 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x478a542a to 127.0.0.1:60416 2023-04-25 06:54:38,393 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:54:38,394 INFO [Listener at localhost/41663] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 06:54:38,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 06:54:38,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:38,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 06:54:38,400 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405678400"}]},"ts":"1682405678400"} 2023-04-25 06:54:38,401 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 06:54:38,403 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 06:54:38,403 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, UNASSIGN}] 2023-04-25 06:54:38,405 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, UNASSIGN 2023-04-25 06:54:38,405 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:38,406 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405678405"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405678405"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405678405"}]},"ts":"1682405678405"} 2023-04-25 06:54:38,406 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:54:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 06:54:38,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:38,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing de84d6c21b0ed9e7c77166f8a0775f87, disabling compactions & flushes 2023-04-25 06:54:38,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:38,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:38,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. after waiting 0 ms 2023-04-25 06:54:38,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:38,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing de84d6c21b0ed9e7c77166f8a0775f87 3/3 column families, dataSize=73.80 KB heapSize=191.63 KB 2023-04-25 06:54:38,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=A 2023-04-25 06:54:38,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:38,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=B 2023-04-25 06:54:38,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:38,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK de84d6c21b0ed9e7c77166f8a0775f87, store=C 2023-04-25 06:54:38,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:38,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 06:54:38,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c7ab34bc76874f22be0f837b11fd55f3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c7ab34bc76874f22be0f837b11fd55f3 2023-04-25 06:54:39,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 06:54:39,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=434, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/84d23db896e5462f96c012c2a77fc67f 2023-04-25 06:54:39,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 06:54:39,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/af5a45a21cbb4df2aeb233fc06abd8a3 2023-04-25 06:54:40,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/ec9123f31f2c4183a9f53c53c2bfa409 2023-04-25 06:54:40,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/A/84d23db896e5462f96c012c2a77fc67f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/84d23db896e5462f96c012c2a77fc67f 2023-04-25 06:54:40,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/84d23db896e5462f96c012c2a77fc67f, entries=150, sequenceid=434, filesize=25.4 K 2023-04-25 06:54:40,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/B/af5a45a21cbb4df2aeb233fc06abd8a3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/af5a45a21cbb4df2aeb233fc06abd8a3 2023-04-25 06:54:40,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/af5a45a21cbb4df2aeb233fc06abd8a3, entries=150, sequenceid=434, filesize=11.9 K 2023-04-25 06:54:40,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/.tmp/C/ec9123f31f2c4183a9f53c53c2bfa409 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/ec9123f31f2c4183a9f53c53c2bfa409 2023-04-25 06:54:40,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/ec9123f31f2c4183a9f53c53c2bfa409, entries=150, sequenceid=434, filesize=11.9 K 2023-04-25 06:54:40,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~191.58 KB/196176, currentSize=0 B/0 for de84d6c21b0ed9e7c77166f8a0775f87 in 1657ms, sequenceid=434, compaction requested=true 2023-04-25 06:54:40,236 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/41456a0a927041e48437b7aaccf85e1c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/32cd490b558646dd90e9cb02f617258d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/beaaf686aecd419195a83db5f17a3e7e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/e21d7d11617d47d59a4596e7533e914d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a0e368d290b4419796920d1370f87acf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5df193c518054c16a09e9f5492eb9c36, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f09578c3d7dd43948975526254778a08, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/69406afb6e5d47fc8526170ae2c98670, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ecbeb98889e243e593c4a7839fdbed32, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f8930b3cd80a44968978a95794700f4f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a5d9d244e57d4cf8895ae6a623023b7c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/c0631597a23040c3b6eb2a67dedc06be, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/34ceeb47824a48ffaf635fec2e50f992, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ffd8446dfade43db9bb50256a58bee6d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a34e469f55a142588bd5f0193cbc6f99, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a4698afa737441e99eb8e636c9161f2a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/498f936d99654c98bbb10e28dc55b098, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/33691276e67641a7959a382b505bc178, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/cb0b1fdf444642f18b6149b87cf9fff8, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5e5ec9f5ced546b38d69b7fc85091aba, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/03f2877d13984b40ab732cf1c744c9da, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/9371d46d1a32438682339312f57e4dfe, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/89fbbc9b375a4c618077fbc1167e9e2a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/fbe322eccc7d498c8d46ab7c26041fb0] to archive 2023-04-25 06:54:40,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:54:40,239 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/41456a0a927041e48437b7aaccf85e1c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/41456a0a927041e48437b7aaccf85e1c 2023-04-25 06:54:40,240 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/32cd490b558646dd90e9cb02f617258d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/32cd490b558646dd90e9cb02f617258d 2023-04-25 06:54:40,241 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/beaaf686aecd419195a83db5f17a3e7e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/beaaf686aecd419195a83db5f17a3e7e 2023-04-25 06:54:40,243 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/e21d7d11617d47d59a4596e7533e914d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/e21d7d11617d47d59a4596e7533e914d 2023-04-25 06:54:40,244 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a0e368d290b4419796920d1370f87acf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a0e368d290b4419796920d1370f87acf 2023-04-25 06:54:40,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5df193c518054c16a09e9f5492eb9c36 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5df193c518054c16a09e9f5492eb9c36 2023-04-25 06:54:40,246 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f09578c3d7dd43948975526254778a08 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f09578c3d7dd43948975526254778a08 2023-04-25 06:54:40,247 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/69406afb6e5d47fc8526170ae2c98670 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/69406afb6e5d47fc8526170ae2c98670 2023-04-25 06:54:40,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ecbeb98889e243e593c4a7839fdbed32 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ecbeb98889e243e593c4a7839fdbed32 2023-04-25 06:54:40,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f8930b3cd80a44968978a95794700f4f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/f8930b3cd80a44968978a95794700f4f 2023-04-25 06:54:40,250 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a5d9d244e57d4cf8895ae6a623023b7c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a5d9d244e57d4cf8895ae6a623023b7c 2023-04-25 06:54:40,251 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/c0631597a23040c3b6eb2a67dedc06be to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/c0631597a23040c3b6eb2a67dedc06be 2023-04-25 06:54:40,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/34ceeb47824a48ffaf635fec2e50f992 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/34ceeb47824a48ffaf635fec2e50f992 2023-04-25 06:54:40,253 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ffd8446dfade43db9bb50256a58bee6d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/ffd8446dfade43db9bb50256a58bee6d 2023-04-25 06:54:40,254 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a34e469f55a142588bd5f0193cbc6f99 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a34e469f55a142588bd5f0193cbc6f99 2023-04-25 06:54:40,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a4698afa737441e99eb8e636c9161f2a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/a4698afa737441e99eb8e636c9161f2a 2023-04-25 06:54:40,256 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/498f936d99654c98bbb10e28dc55b098 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/498f936d99654c98bbb10e28dc55b098 2023-04-25 06:54:40,257 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/33691276e67641a7959a382b505bc178 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/33691276e67641a7959a382b505bc178 2023-04-25 06:54:40,258 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/cb0b1fdf444642f18b6149b87cf9fff8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/cb0b1fdf444642f18b6149b87cf9fff8 2023-04-25 06:54:40,259 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5e5ec9f5ced546b38d69b7fc85091aba to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/5e5ec9f5ced546b38d69b7fc85091aba 2023-04-25 06:54:40,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/03f2877d13984b40ab732cf1c744c9da to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/03f2877d13984b40ab732cf1c744c9da 2023-04-25 06:54:40,261 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/9371d46d1a32438682339312f57e4dfe to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/9371d46d1a32438682339312f57e4dfe 2023-04-25 06:54:40,262 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/89fbbc9b375a4c618077fbc1167e9e2a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/89fbbc9b375a4c618077fbc1167e9e2a 2023-04-25 06:54:40,263 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/fbe322eccc7d498c8d46ab7c26041fb0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/fbe322eccc7d498c8d46ab7c26041fb0 2023-04-25 06:54:40,266 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b6e044b6347b43cfaa3ba1b5c5ac6530, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/12079431d8314b68a4b587d481cca8b6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8d15340ac00b4d42826ae79dc4f78236, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/99cfc7ab27574c58bf3cee635ee85a34, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/dfdb2b8600eb4872ac614a00f29c8b7b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/d7cfa5b1ad4345a08749fa38953cba5a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/322184d7ed4f42f7921f321229e526aa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b64deabbd0b643d8a0a963b553676b9f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/270e71513fba4d588b93405a6bb79b9c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/67a0389ecd74424eab4716016cbdf175, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e4ae318ca0444bf6ba6ecb58cbf8292f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fec28c6f7ae54633a84fa21419bbf428, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fab318e413334aae82befb9123f23a45, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f27ea0cf967846d58a1faad589debf12, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/29a279d207ff46b8855cb5be417044c6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b3bfe5ad6f55401cb93b3e094c231c2b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/3be026220cbb400a86c1c0f6f80481ca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/324709ec0ca54ffeb12977c3a6b73de4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/37a95e6c4dc040f3a15e06da5540ed91, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/ade6a286ed4b46dfad6d7506cad6c51b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/da16b9ff3a1f4b12a58868dfbef0f22f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8914cc5b62934c678eb260929ad8b7f5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/74256f362b1c4f25af96e8299ee154c8] to archive 2023-04-25 06:54:40,266 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:54:40,267 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b6e044b6347b43cfaa3ba1b5c5ac6530 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b6e044b6347b43cfaa3ba1b5c5ac6530 2023-04-25 06:54:40,268 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/12079431d8314b68a4b587d481cca8b6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/12079431d8314b68a4b587d481cca8b6 2023-04-25 06:54:40,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8d15340ac00b4d42826ae79dc4f78236 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8d15340ac00b4d42826ae79dc4f78236 2023-04-25 06:54:40,270 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/99cfc7ab27574c58bf3cee635ee85a34 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/99cfc7ab27574c58bf3cee635ee85a34 2023-04-25 06:54:40,271 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/dfdb2b8600eb4872ac614a00f29c8b7b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/dfdb2b8600eb4872ac614a00f29c8b7b 2023-04-25 06:54:40,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/d7cfa5b1ad4345a08749fa38953cba5a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/d7cfa5b1ad4345a08749fa38953cba5a 2023-04-25 06:54:40,273 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/322184d7ed4f42f7921f321229e526aa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/322184d7ed4f42f7921f321229e526aa 2023-04-25 06:54:40,274 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b64deabbd0b643d8a0a963b553676b9f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b64deabbd0b643d8a0a963b553676b9f 2023-04-25 06:54:40,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/270e71513fba4d588b93405a6bb79b9c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/270e71513fba4d588b93405a6bb79b9c 2023-04-25 06:54:40,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/67a0389ecd74424eab4716016cbdf175 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/67a0389ecd74424eab4716016cbdf175 2023-04-25 06:54:40,277 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e4ae318ca0444bf6ba6ecb58cbf8292f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e4ae318ca0444bf6ba6ecb58cbf8292f 2023-04-25 06:54:40,278 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fec28c6f7ae54633a84fa21419bbf428 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fec28c6f7ae54633a84fa21419bbf428 2023-04-25 06:54:40,279 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fab318e413334aae82befb9123f23a45 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/fab318e413334aae82befb9123f23a45 2023-04-25 06:54:40,281 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f27ea0cf967846d58a1faad589debf12 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f27ea0cf967846d58a1faad589debf12 2023-04-25 06:54:40,281 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/29a279d207ff46b8855cb5be417044c6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/29a279d207ff46b8855cb5be417044c6 2023-04-25 06:54:40,282 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b3bfe5ad6f55401cb93b3e094c231c2b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/b3bfe5ad6f55401cb93b3e094c231c2b 2023-04-25 06:54:40,283 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/e09ebe5e84804c2ab2f8ab3d15c1aa3c 2023-04-25 06:54:40,284 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/3be026220cbb400a86c1c0f6f80481ca to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/3be026220cbb400a86c1c0f6f80481ca 2023-04-25 06:54:40,285 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/324709ec0ca54ffeb12977c3a6b73de4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/324709ec0ca54ffeb12977c3a6b73de4 2023-04-25 06:54:40,286 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/37a95e6c4dc040f3a15e06da5540ed91 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/37a95e6c4dc040f3a15e06da5540ed91 2023-04-25 06:54:40,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/ade6a286ed4b46dfad6d7506cad6c51b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/ade6a286ed4b46dfad6d7506cad6c51b 2023-04-25 06:54:40,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/da16b9ff3a1f4b12a58868dfbef0f22f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/da16b9ff3a1f4b12a58868dfbef0f22f 2023-04-25 06:54:40,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8914cc5b62934c678eb260929ad8b7f5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/8914cc5b62934c678eb260929ad8b7f5 2023-04-25 06:54:40,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/74256f362b1c4f25af96e8299ee154c8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/74256f362b1c4f25af96e8299ee154c8 2023-04-25 06:54:40,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/f4a56b446cb242d092af769979574806, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/728c22f1fda843cb8bb7ac24a77b6355, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/acadb9396f3043b0849335155930a213, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e1c9e5279ae243cba568b31771707d92, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/93249af5659643c98fc52d4396dbc9dc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e8ef650e5f8e47b99f463189d6c79aaa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fc2f49cbd0ee43d7bee804672b03051f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9b3d972d2d44af28a82452447c6fbbc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/baed3152b7e24cf48260899f7546988b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/5c5752d188ff48d698ea1b512deb3844, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/077bd8cbc86348a2b9f524b4f9e96624, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/9aea241502c44fe7927c4b50a340c4f0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/19ef959183584f059961b9efec612647, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/0bea725a0f4c49dc801ebce40e17f3cb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1a3962cbb3694c008662423497f0fc74, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/3ece4f001f9540ff9b4f05c21526d4d2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/2a32678bd5a949ab8f912c53de15b101, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fca9c55aa01241a1b9e8e9d8f4df4d13, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/786dc170579b4e4ba77ff5534e17e830, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/c3f62d9f0dad4987bf61d68a6132f05c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/369ba48778a14468b91be12def320216, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/4538c9667e0c4b69b458dcfb7fc83e0f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/29c7d09292d2446f8e843e39dc68a6e9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1233d48411924ea5a0ba8cb2f3901bd3] to archive 2023-04-25 06:54:40,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:54:40,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/f4a56b446cb242d092af769979574806 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/f4a56b446cb242d092af769979574806 2023-04-25 06:54:40,295 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/728c22f1fda843cb8bb7ac24a77b6355 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/728c22f1fda843cb8bb7ac24a77b6355 2023-04-25 06:54:40,296 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/acadb9396f3043b0849335155930a213 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/acadb9396f3043b0849335155930a213 2023-04-25 06:54:40,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e1c9e5279ae243cba568b31771707d92 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e1c9e5279ae243cba568b31771707d92 2023-04-25 06:54:40,298 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/93249af5659643c98fc52d4396dbc9dc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/93249af5659643c98fc52d4396dbc9dc 2023-04-25 06:54:40,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e8ef650e5f8e47b99f463189d6c79aaa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/e8ef650e5f8e47b99f463189d6c79aaa 2023-04-25 06:54:40,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fc2f49cbd0ee43d7bee804672b03051f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fc2f49cbd0ee43d7bee804672b03051f 2023-04-25 06:54:40,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9b3d972d2d44af28a82452447c6fbbc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9b3d972d2d44af28a82452447c6fbbc 2023-04-25 06:54:40,302 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/baed3152b7e24cf48260899f7546988b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/baed3152b7e24cf48260899f7546988b 2023-04-25 06:54:40,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/5c5752d188ff48d698ea1b512deb3844 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/5c5752d188ff48d698ea1b512deb3844 2023-04-25 06:54:40,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/077bd8cbc86348a2b9f524b4f9e96624 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/077bd8cbc86348a2b9f524b4f9e96624 2023-04-25 06:54:40,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/9aea241502c44fe7927c4b50a340c4f0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/9aea241502c44fe7927c4b50a340c4f0 2023-04-25 06:54:40,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/19ef959183584f059961b9efec612647 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/19ef959183584f059961b9efec612647 2023-04-25 06:54:40,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/0bea725a0f4c49dc801ebce40e17f3cb to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/0bea725a0f4c49dc801ebce40e17f3cb 2023-04-25 06:54:40,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1a3962cbb3694c008662423497f0fc74 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1a3962cbb3694c008662423497f0fc74 2023-04-25 06:54:40,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/3ece4f001f9540ff9b4f05c21526d4d2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/3ece4f001f9540ff9b4f05c21526d4d2 2023-04-25 06:54:40,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/2a32678bd5a949ab8f912c53de15b101 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/2a32678bd5a949ab8f912c53de15b101 2023-04-25 06:54:40,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fca9c55aa01241a1b9e8e9d8f4df4d13 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/fca9c55aa01241a1b9e8e9d8f4df4d13 2023-04-25 06:54:40,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/786dc170579b4e4ba77ff5534e17e830 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/786dc170579b4e4ba77ff5534e17e830 2023-04-25 06:54:40,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/c3f62d9f0dad4987bf61d68a6132f05c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/c3f62d9f0dad4987bf61d68a6132f05c 2023-04-25 06:54:40,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/369ba48778a14468b91be12def320216 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/369ba48778a14468b91be12def320216 2023-04-25 06:54:40,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/4538c9667e0c4b69b458dcfb7fc83e0f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/4538c9667e0c4b69b458dcfb7fc83e0f 2023-04-25 06:54:40,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/29c7d09292d2446f8e843e39dc68a6e9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/29c7d09292d2446f8e843e39dc68a6e9 2023-04-25 06:54:40,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1233d48411924ea5a0ba8cb2f3901bd3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/1233d48411924ea5a0ba8cb2f3901bd3 2023-04-25 06:54:40,320 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/recovered.edits/437.seqid, newMaxSeqId=437, maxSeqId=4 2023-04-25 06:54:40,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87. 2023-04-25 06:54:40,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for de84d6c21b0ed9e7c77166f8a0775f87: 2023-04-25 06:54:40,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:40,323 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=de84d6c21b0ed9e7c77166f8a0775f87, regionState=CLOSED 2023-04-25 06:54:40,323 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405680323"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405680323"}]},"ts":"1682405680323"} 2023-04-25 06:54:40,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-25 06:54:40,326 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure de84d6c21b0ed9e7c77166f8a0775f87, server=jenkins-hbase4.apache.org,41021,1682405622759 in 1.9180 sec 2023-04-25 06:54:40,327 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-25 06:54:40,327 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=de84d6c21b0ed9e7c77166f8a0775f87, UNASSIGN in 1.9230 sec 2023-04-25 06:54:40,328 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405680327"}]},"ts":"1682405680327"} 2023-04-25 06:54:40,328 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 06:54:40,331 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 06:54:40,332 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9360 sec 2023-04-25 06:54:40,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 06:54:40,504 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-25 06:54:40,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 06:54:40,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:40,507 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:40,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-25 06:54:40,507 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:40,512 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:40,514 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/recovered.edits] 2023-04-25 06:54:40,517 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/84d23db896e5462f96c012c2a77fc67f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/84d23db896e5462f96c012c2a77fc67f 2023-04-25 06:54:40,519 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/b608d05ecd3f4c80acc81adc4c85600a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/b608d05ecd3f4c80acc81adc4c85600a 2023-04-25 06:54:40,520 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/d5cab30d1624441797fc26547ebb1021 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/A/d5cab30d1624441797fc26547ebb1021 2023-04-25 06:54:40,523 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/6d82821cf1e24872833a7f7b10de6ece to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/6d82821cf1e24872833a7f7b10de6ece 2023-04-25 06:54:40,524 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/af5a45a21cbb4df2aeb233fc06abd8a3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/af5a45a21cbb4df2aeb233fc06abd8a3 2023-04-25 06:54:40,525 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f0ca6c56d3224b1aa216a827c898590b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/B/f0ca6c56d3224b1aa216a827c898590b 2023-04-25 06:54:40,528 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9d063d1e1834ad999ded65155c677d2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/a9d063d1e1834ad999ded65155c677d2 2023-04-25 06:54:40,529 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/dc201d2b99f9427e9f5e741e67c3ea08 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/dc201d2b99f9427e9f5e741e67c3ea08 2023-04-25 06:54:40,531 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/ec9123f31f2c4183a9f53c53c2bfa409 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/C/ec9123f31f2c4183a9f53c53c2bfa409 2023-04-25 06:54:40,534 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/recovered.edits/437.seqid to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87/recovered.edits/437.seqid 2023-04-25 06:54:40,534 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/de84d6c21b0ed9e7c77166f8a0775f87 2023-04-25 06:54:40,534 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:54:40,535 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 06:54:40,536 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 06:54:40,539 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250e6a392c358c4c078917f7fbfd7b9dbe to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250e6a392c358c4c078917f7fbfd7b9dbe 2023-04-25 06:54:40,541 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251432c143c55d4bcfbaddc79c7ece09f0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251432c143c55d4bcfbaddc79c7ece09f0 2023-04-25 06:54:40,542 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254884f98d886a46069eca2be5f5cdb2d6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254884f98d886a46069eca2be5f5cdb2d6 2023-04-25 06:54:40,543 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042549729fad9ab140ebb350fcca518036d0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042549729fad9ab140ebb350fcca518036d0 2023-04-25 06:54:40,544 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425514267140c4d4b048223c1db8746826a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425514267140c4d4b048223c1db8746826a 2023-04-25 06:54:40,545 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255554a0fffeb34458b7cf04af86280028 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255554a0fffeb34458b7cf04af86280028 2023-04-25 06:54:40,546 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255723bb0a1c03467bab5133472e5b95dd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255723bb0a1c03467bab5133472e5b95dd 2023-04-25 06:54:40,547 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425589ca9d6bbf54a3596a3aed494c6534a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425589ca9d6bbf54a3596a3aed494c6534a 2023-04-25 06:54:40,548 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042564591901391644359ff5b0c5b8cbcedf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042564591901391644359ff5b0c5b8cbcedf 2023-04-25 06:54:40,549 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257692501dff26441bb1505abf1db43fbd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257692501dff26441bb1505abf1db43fbd 2023-04-25 06:54:40,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042585cb66d4fc50496d82b12399a37e85d0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042585cb66d4fc50496d82b12399a37e85d0 2023-04-25 06:54:40,551 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a7b4763ef9e64862a7c7412b6661368a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a7b4763ef9e64862a7c7412b6661368a 2023-04-25 06:54:40,552 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b21bfe4debe94ec39c88131dc509c63b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b21bfe4debe94ec39c88131dc509c63b 2023-04-25 06:54:40,553 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c122df2ffbc84038b40cd92cc787f53f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c122df2ffbc84038b40cd92cc787f53f 2023-04-25 06:54:40,554 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c4ed974fbe4b40b78e67f438fe29ee23 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c4ed974fbe4b40b78e67f438fe29ee23 2023-04-25 06:54:40,555 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c7ab34bc76874f22be0f837b11fd55f3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c7ab34bc76874f22be0f837b11fd55f3 2023-04-25 06:54:40,556 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cab2cdc32abd46bf9647384c7e2eca35 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cab2cdc32abd46bf9647384c7e2eca35 2023-04-25 06:54:40,557 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cd161127a6c9403ba50a90ebd1e241b4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cd161127a6c9403ba50a90ebd1e241b4 2023-04-25 06:54:40,558 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ce9c524d600f43d7b3ba3474f4062b3c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ce9c524d600f43d7b3ba3474f4062b3c 2023-04-25 06:54:40,559 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425daf51c83f2dd406cbb10486571f4af25 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425daf51c83f2dd406cbb10486571f4af25 2023-04-25 06:54:40,560 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425de8ce85419f244f1b98e69b8ca3504a0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425de8ce85419f244f1b98e69b8ca3504a0 2023-04-25 06:54:40,561 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ea5b5094b8fa434f8d44e5cb6afe6ce4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ea5b5094b8fa434f8d44e5cb6afe6ce4 2023-04-25 06:54:40,561 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 06:54:40,563 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:40,565 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 06:54:40,567 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 06:54:40,568 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:40,568 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 06:54:40,568 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682405680568"}]},"ts":"9223372036854775807"} 2023-04-25 06:54:40,570 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 06:54:40,570 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => de84d6c21b0ed9e7c77166f8a0775f87, NAME => 'TestAcidGuarantees,,1682405654019.de84d6c21b0ed9e7c77166f8a0775f87.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 06:54:40,570 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 06:54:40,570 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682405680570"}]},"ts":"9223372036854775807"} 2023-04-25 06:54:40,571 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 06:54:40,573 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:40,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 68 msec 2023-04-25 06:54:40,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-25 06:54:40,608 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-25 06:54:40,621 INFO [Listener at localhost/41663] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235 (was 234) - Thread LEAK? -, OpenFileDescriptor=555 (was 561), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=442 (was 386) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=4302 (was 4382) 2023-04-25 06:54:40,633 INFO [Listener at localhost/41663] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=235, OpenFileDescriptor=555, MaxFileDescriptor=60000, SystemLoadAverage=442, ProcessCount=172, AvailableMemoryMB=4301 2023-04-25 06:54:40,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:54:40,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:54:40,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 06:54:40,638 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 06:54:40,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-25 06:54:40,639 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 06:54:40,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 06:54:40,641 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:40,642 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17 empty. 2023-04-25 06:54:40,642 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:40,642 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:54:40,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 06:54:40,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 06:54:41,052 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:54:41,054 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 11919fcb635bcaf9ea0c7782550d9d17, NAME => 'TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp 2023-04-25 06:54:41,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 06:54:41,460 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:54:41,460 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 11919fcb635bcaf9ea0c7782550d9d17, disabling compactions & flushes 2023-04-25 06:54:41,460 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,460 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,461 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. after waiting 1 ms 2023-04-25 06:54:41,461 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,461 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,461 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:41,463 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 06:54:41,464 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405681464"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405681464"}]},"ts":"1682405681464"} 2023-04-25 06:54:41,465 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 06:54:41,466 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 06:54:41,466 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405681466"}]},"ts":"1682405681466"} 2023-04-25 06:54:41,467 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 06:54:41,469 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=11919fcb635bcaf9ea0c7782550d9d17, ASSIGN}] 2023-04-25 06:54:41,471 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=11919fcb635bcaf9ea0c7782550d9d17, ASSIGN 2023-04-25 06:54:41,472 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=11919fcb635bcaf9ea0c7782550d9d17, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=false 2023-04-25 06:54:41,623 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=11919fcb635bcaf9ea0c7782550d9d17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:41,623 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405681623"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405681623"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405681623"}]},"ts":"1682405681623"} 2023-04-25 06:54:41,624 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:54:41,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 06:54:41,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 11919fcb635bcaf9ea0c7782550d9d17, NAME => 'TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:54:41,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:54:41,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,781 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,782 DEBUG [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A 2023-04-25 06:54:41,782 DEBUG [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A 2023-04-25 06:54:41,782 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:41,783 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11919fcb635bcaf9ea0c7782550d9d17 columnFamilyName A 2023-04-25 06:54:41,783 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.HStore(310): Store=11919fcb635bcaf9ea0c7782550d9d17/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:41,783 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,784 DEBUG [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B 2023-04-25 06:54:41,784 DEBUG [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B 2023-04-25 06:54:41,785 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:41,785 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11919fcb635bcaf9ea0c7782550d9d17 columnFamilyName B 2023-04-25 06:54:41,785 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.HStore(310): Store=11919fcb635bcaf9ea0c7782550d9d17/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:41,785 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,786 DEBUG [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C 2023-04-25 06:54:41,787 DEBUG [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C 2023-04-25 06:54:41,787 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:54:41,787 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11919fcb635bcaf9ea0c7782550d9d17 columnFamilyName C 2023-04-25 06:54:41,787 INFO [StoreOpener-11919fcb635bcaf9ea0c7782550d9d17-1] regionserver.HStore(310): Store=11919fcb635bcaf9ea0c7782550d9d17/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:54:41,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:54:41,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:41,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:54:41,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 11919fcb635bcaf9ea0c7782550d9d17; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64174416, jitterRate=-0.04372668266296387}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:54:41,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:41,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., pid=30, masterSystemTime=1682405681775 2023-04-25 06:54:41,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:41,796 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=11919fcb635bcaf9ea0c7782550d9d17, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:41,796 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405681796"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405681796"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405681796"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405681796"}]},"ts":"1682405681796"} 2023-04-25 06:54:41,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-25 06:54:41,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 in 173 msec 2023-04-25 06:54:41,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-25 06:54:41,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=11919fcb635bcaf9ea0c7782550d9d17, ASSIGN in 329 msec 2023-04-25 06:54:41,800 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 06:54:41,801 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405681800"}]},"ts":"1682405681800"} 2023-04-25 06:54:41,801 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 06:54:41,805 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 06:54:41,806 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1690 sec 2023-04-25 06:54:42,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 06:54:42,745 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-25 06:54:42,746 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x53921272 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,751 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7cdc2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,752 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,753 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51272, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,754 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 06:54:42,756 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 06:54:42,757 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x13544701 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,762 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5096c69b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,762 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x10b79c2c to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,766 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@216aad74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,766 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5a81fa to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,772 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76203061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,772 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x3562c7bf to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,776 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b39d5b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,776 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x6705d115 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,779 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44f9bd75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,780 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x5e5b1c14 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,783 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574000bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,784 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4f29c1c4 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,791 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4994df89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,791 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x681ace6c to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,795 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69bc154a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,795 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x2b4fa5d3 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,798 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15186778, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,799 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x45de5ad8 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:54:42,802 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e90e8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:54:42,806 DEBUG [hconnection-0x49831cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,806 DEBUG [hconnection-0x4e679d0e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,806 DEBUG [hconnection-0x50fd84a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,807 DEBUG [hconnection-0x35e96757-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,807 DEBUG [hconnection-0x7bceab06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,808 DEBUG [hconnection-0x45c7962f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:42,808 DEBUG [hconnection-0x639dd0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,808 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,809 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,813 DEBUG [hconnection-0x6cf0263-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,817 DEBUG [hconnection-0x3491eb19-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,817 DEBUG [hconnection-0x72fb5ea1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:54:42,818 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,819 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,819 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,819 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,820 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,822 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,826 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,826 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:54:42,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:42,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:42,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:42,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:54:42,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:42,846 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:42,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:42,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:42,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:42,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:42,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:42,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:42,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:42,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:42,848 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,848 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,855 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:42,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:42,855 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,856 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:42,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:42,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:42,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:42,857 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:42,857 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:42,857 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:42,857 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:42,859 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:42,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:42,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:42,861 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:42,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@729462dd[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:42,861 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:42,862 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,862 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:42,863 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:42,863 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,863 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:42,864 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:42,866 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:42,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:42,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:42,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:42,866 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:42,868 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,868 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:42,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:42,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:42,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:42,869 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:42,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:42,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:42,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:42,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:42,872 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:42,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,873 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,873 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,873 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,873 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,873 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:42,873 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:42,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405742912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:42,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:42,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405742914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405742914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:42,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:42,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405742916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405742916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,973 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:42,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:42,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:42,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:42,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:42,989 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:42,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:42,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:42,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:42,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,991 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:42,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:42,991 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,992 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,992 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:42,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:42,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:42,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:42,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:42,993 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:42,993 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:42,993 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:42,994 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:42,994 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:42,994 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:42,994 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:42,995 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:42,996 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,996 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:42,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:42,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d781f1[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:42,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:42,997 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,999 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:42,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,000 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,000 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:43,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,004 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,004 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,005 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:43,007 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:43,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:43,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:43,007 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:43,007 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:43,009 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,009 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,009 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,009 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,013 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,013 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,013 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,013 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,013 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,013 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,013 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:43,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:43,014 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:43,014 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:43,014 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:43,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,014 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,014 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405743017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405743018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405743018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405743019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405743019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,114 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:43,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:43,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:43,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:43,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:43,117 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:43,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:43,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:43,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,118 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,120 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:43,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:43,121 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,122 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:43,122 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:43,122 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:43,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:43,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:43,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:43,123 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,123 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,124 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,124 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@410f679e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:43,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:43,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,126 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:43,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:43,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:43,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:43,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:43,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:43,130 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:43,131 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:43,131 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:43,131 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:43,131 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:43,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,134 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:43,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,138 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:43,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,138 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:43,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405743219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405743221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405743221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405743222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405743222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,238 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:43,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:43,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:43,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:43,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:43,242 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:43,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:43,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:43,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,243 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,243 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:43,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:43,245 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,245 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:43,245 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:43,246 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:43,247 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68cd7cce[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:43,249 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:43,249 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:43,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:43,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:43,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:43,253 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:43,255 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:43,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:43,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:43,255 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:43,255 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:43,257 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,257 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,257 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,258 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,259 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,261 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:43,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:43,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:43,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,262 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,262 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,262 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:43,262 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:43,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/5598acc8c8b543579e3861912dfde771 2023-04-25 06:54:43,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/2a6a5c8d34c34d2fbd6e0de8af5e363c 2023-04-25 06:54:43,362 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:43,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:43,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:43,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:43,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:43,365 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:43,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:43,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:43,366 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,366 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,367 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:43,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:43,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:43,369 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,369 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:43,369 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:43,369 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:43,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,371 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:43,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391e6980[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:43,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:43,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,374 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,374 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:43,374 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:43,374 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:43,374 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,374 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:43,375 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:43,378 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:43,378 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:43,378 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:43,378 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:43,378 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:43,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,383 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,387 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,387 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:43,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,387 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,387 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,388 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:43,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,388 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:43,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,388 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,488 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:43,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:43,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:43,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:43,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:43,491 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:43,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:43,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:43,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,493 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,493 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,493 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,493 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:43,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:43,495 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,495 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:43,495 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:43,495 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:43,502 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51aeaf5b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:43,503 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:43,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,505 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,505 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:43,505 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:43,505 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:43,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:43,506 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:43,506 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,506 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:43,506 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:43,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:43,509 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:43,509 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:43,509 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:43,509 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:43,509 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:43,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,511 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,511 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,512 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:43,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:43,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,517 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,517 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:43,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,517 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405743523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405743525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405743525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405743526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:43,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405743527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,617 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:43,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:43,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:43,620 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:43,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:43,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:43,621 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,621 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,622 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,622 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,622 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,622 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:43,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:43,624 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,624 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:43,624 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:43,624 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:43,625 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,626 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b271167[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:43,627 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:43,627 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,628 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,629 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:43,629 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:43,629 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:43,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:43,631 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:43,633 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:43,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:43,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:43,634 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:43,634 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:43,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:43,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,636 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,637 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:43,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:43,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:43,640 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:43,640 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:43,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:43,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,641 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:43,641 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/cebf695a4ce74faca3a29d17f51f4430 2023-04-25 06:54:43,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/5598acc8c8b543579e3861912dfde771 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/5598acc8c8b543579e3861912dfde771 2023-04-25 06:54:43,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/5598acc8c8b543579e3861912dfde771, entries=150, sequenceid=14, filesize=11.6 K 2023-04-25 06:54:43,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/2a6a5c8d34c34d2fbd6e0de8af5e363c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2a6a5c8d34c34d2fbd6e0de8af5e363c 2023-04-25 06:54:43,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2a6a5c8d34c34d2fbd6e0de8af5e363c, entries=150, sequenceid=14, filesize=11.6 K 2023-04-25 06:54:43,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/cebf695a4ce74faca3a29d17f51f4430 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cebf695a4ce74faca3a29d17f51f4430 2023-04-25 06:54:43,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cebf695a4ce74faca3a29d17f51f4430, entries=150, sequenceid=14, filesize=11.6 K 2023-04-25 06:54:43,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 11919fcb635bcaf9ea0c7782550d9d17 in 904ms, sequenceid=14, compaction requested=false 2023-04-25 06:54:43,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:43,740 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:43,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:43,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:43,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:43,744 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:43,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:43,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:43,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,746 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:43,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:43,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:43,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:43,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:43,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:43,748 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:43,748 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:43,748 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:43,748 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:43,748 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:43,748 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:43,748 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:43,748 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:54:43,748 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:43,748 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:43,748 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:43,748 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:43,749 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:43,749 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:43,776 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/8d409beef1de41df9a48a026065bd337 2023-04-25 06:54:43,814 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9137b6a8468f46ca8f872f77e02af7fe 2023-04-25 06:54:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:44,054 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:44,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405744059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405744059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405744060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405744062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405744063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405744164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405744164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405744164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405744166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405744166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,243 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/75c31984e70c4d8abb71f798b30a4f55 2023-04-25 06:54:44,250 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/8d409beef1de41df9a48a026065bd337 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/8d409beef1de41df9a48a026065bd337 2023-04-25 06:54:44,258 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/8d409beef1de41df9a48a026065bd337, entries=150, sequenceid=34, filesize=11.6 K 2023-04-25 06:54:44,259 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9137b6a8468f46ca8f872f77e02af7fe as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9137b6a8468f46ca8f872f77e02af7fe 2023-04-25 06:54:44,263 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9137b6a8468f46ca8f872f77e02af7fe, entries=150, sequenceid=34, filesize=11.6 K 2023-04-25 06:54:44,264 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/75c31984e70c4d8abb71f798b30a4f55 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/75c31984e70c4d8abb71f798b30a4f55 2023-04-25 06:54:44,268 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/75c31984e70c4d8abb71f798b30a4f55, entries=150, sequenceid=34, filesize=11.6 K 2023-04-25 06:54:44,269 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 11919fcb635bcaf9ea0c7782550d9d17 in 521ms, sequenceid=34, compaction requested=false 2023-04-25 06:54:44,269 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:44,269 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:44,271 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,271 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688f3bc1[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:44,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:44,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:44,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:44,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:44,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:44,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:44,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:44,281 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:44,281 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:44,281 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:44,281 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:44,281 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:44,283 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,283 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,283 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,284 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,291 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:44,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:44,291 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:44,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,292 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,292 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:44,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:44,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:54:44,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:44,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:44,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:44,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:44,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:44,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:44,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/696bacb104c94672a3de01056818bc08 2023-04-25 06:54:44,392 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:44,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405744389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405744392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:44,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:44,396 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:44,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:44,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:44,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405744394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,398 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405744395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405744395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:44,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:44,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:44,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:44,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:44,423 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,423 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:44,423 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:44,423 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,423 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:44,423 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:44,423 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,423 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:44,423 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,425 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:44,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:44,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22971d39[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:44,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:44,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/7223b3c8085e4e8b8a05f2e100b78192 2023-04-25 06:54:44,429 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,429 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:44,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:44,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:44,430 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,430 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:44,431 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:44,431 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,431 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,431 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:44,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:44,435 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:44,435 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:44,435 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:44,435 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:44,435 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:44,437 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,437 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,437 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,438 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,439 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,448 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:44,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:44,449 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:44,449 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:44,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405744495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405744496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405744499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405744500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405744500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,549 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:44,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:44,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:44,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:44,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:44,552 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:44,552 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,552 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:44,552 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:44,553 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,553 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,558 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,558 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,558 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,558 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:44,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:44,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:44,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:44,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:44,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:44,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e7aa47[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:44,563 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:44,563 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,565 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,565 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,565 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:44,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:44,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:44,567 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:44,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:44,571 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:44,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:44,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:44,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:44,571 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:44,574 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,574 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,574 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,576 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:44,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:44,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:44,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,579 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:44,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,579 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,679 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:44,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:44,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:44,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:44,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:44,682 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:44,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:44,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:44,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:44,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:44,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:44,687 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,687 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:44,688 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:44,688 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:44,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:44,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:44,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:44,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:44,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55182fd5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:44,692 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:44,693 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:44,695 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,695 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:44,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,697 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:44,697 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,697 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:44,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:44,699 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:44,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:44,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:44,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:44,699 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:44,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405744698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405744698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,702 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405744701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405744703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:44,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405744703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,710 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:44,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:44,710 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,710 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:44,710 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:44,710 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,710 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,710 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,710 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,710 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,810 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:44,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:44,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:44,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:44,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:44,813 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:44,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:44,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:44,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,815 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:44,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:44,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:44,817 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,817 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:44,817 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:44,817 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:44,818 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,819 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,819 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:44,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d84784[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:44,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:44,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:44,821 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:44,821 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:44,821 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:44,823 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:44,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:44,826 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:44,826 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:44,826 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:44,826 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:44,826 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:44,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:44,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:44,828 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,832 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,832 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,832 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:44,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:44,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:44,832 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:44,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,833 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:44,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:44,833 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:44,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:44,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/fdfbe714f1e94789820aac60da73d172 2023-04-25 06:54:44,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/696bacb104c94672a3de01056818bc08 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/696bacb104c94672a3de01056818bc08 2023-04-25 06:54:44,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/696bacb104c94672a3de01056818bc08, entries=250, sequenceid=50, filesize=16.2 K 2023-04-25 06:54:44,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/7223b3c8085e4e8b8a05f2e100b78192 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7223b3c8085e4e8b8a05f2e100b78192 2023-04-25 06:54:44,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7223b3c8085e4e8b8a05f2e100b78192, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 06:54:44,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/fdfbe714f1e94789820aac60da73d172 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/fdfbe714f1e94789820aac60da73d172 2023-04-25 06:54:44,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/fdfbe714f1e94789820aac60da73d172, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 06:54:44,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 11919fcb635bcaf9ea0c7782550d9d17 in 509ms, sequenceid=50, compaction requested=true 2023-04-25 06:54:44,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:44,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:44,878 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:44,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:44,878 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:44,879 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:44,880 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:44,880 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/A is initiating minor compaction (all files) 2023-04-25 06:54:44,880 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:44,880 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/A in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,880 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/B is initiating minor compaction (all files) 2023-04-25 06:54:44,880 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/5598acc8c8b543579e3861912dfde771, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/8d409beef1de41df9a48a026065bd337, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/696bacb104c94672a3de01056818bc08] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=39.4 K 2023-04-25 06:54:44,880 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/B in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,880 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2a6a5c8d34c34d2fbd6e0de8af5e363c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9137b6a8468f46ca8f872f77e02af7fe, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7223b3c8085e4e8b8a05f2e100b78192] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=34.8 K 2023-04-25 06:54:44,881 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5598acc8c8b543579e3861912dfde771, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682405682830 2023-04-25 06:54:44,881 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2a6a5c8d34c34d2fbd6e0de8af5e363c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682405682830 2023-04-25 06:54:44,881 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 8d409beef1de41df9a48a026065bd337, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682405682910 2023-04-25 06:54:44,882 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9137b6a8468f46ca8f872f77e02af7fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682405682910 2023-04-25 06:54:44,882 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 696bacb104c94672a3de01056818bc08, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405684061 2023-04-25 06:54:44,882 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7223b3c8085e4e8b8a05f2e100b78192, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405684061 2023-04-25 06:54:44,897 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#B#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:44,897 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#A#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:44,908 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/3c16d57075564c60989b4c5afff20dce as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c16d57075564c60989b4c5afff20dce 2023-04-25 06:54:44,913 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/B of 11919fcb635bcaf9ea0c7782550d9d17 into 3c16d57075564c60989b4c5afff20dce(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:44,913 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:44,913 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/B, priority=13, startTime=1682405684878; duration=0sec 2023-04-25 06:54:44,914 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:44,914 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:44,915 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:44,915 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/C is initiating minor compaction (all files) 2023-04-25 06:54:44,915 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/C in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,915 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cebf695a4ce74faca3a29d17f51f4430, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/75c31984e70c4d8abb71f798b30a4f55, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/fdfbe714f1e94789820aac60da73d172] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=34.8 K 2023-04-25 06:54:44,916 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting cebf695a4ce74faca3a29d17f51f4430, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682405682830 2023-04-25 06:54:44,916 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 75c31984e70c4d8abb71f798b30a4f55, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682405682910 2023-04-25 06:54:44,917 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting fdfbe714f1e94789820aac60da73d172, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405684061 2023-04-25 06:54:44,926 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#C#compaction#200 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:44,933 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:44,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:44,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:44,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:44,948 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:44,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:44,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:44,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:44,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:44,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:44,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:44,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:44,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:44,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:44,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:44,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:44,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:44,961 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:44,961 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:44,962 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:54:44,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:44,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:44,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:44,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:44,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:44,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:45,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:45,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:45,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405745013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405745014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405745015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405745016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405745016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405745117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405745118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405745119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405745119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405745119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,311 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/c72b299468af4d0eb466fe4143e07293 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c72b299468af4d0eb466fe4143e07293 2023-04-25 06:54:45,318 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/A of 11919fcb635bcaf9ea0c7782550d9d17 into c72b299468af4d0eb466fe4143e07293(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:45,318 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:45,318 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/A, priority=13, startTime=1682405684878; duration=0sec 2023-04-25 06:54:45,318 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:45,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405745320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405745323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405745324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405745333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405745337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,351 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/bc10b4c54e8a4627a5e2321ec36e6b66 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/bc10b4c54e8a4627a5e2321ec36e6b66 2023-04-25 06:54:45,356 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/C of 11919fcb635bcaf9ea0c7782550d9d17 into bc10b4c54e8a4627a5e2321ec36e6b66(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:45,356 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:45,356 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/C, priority=13, startTime=1682405684879; duration=0sec 2023-04-25 06:54:45,356 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:45,374 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/e84b366603264f0d82fc7656650f4809 2023-04-25 06:54:45,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405745626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405745628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405745629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405745636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:45,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405745638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:45,784 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e52cbc39862f4846a76498bdde7095f7 2023-04-25 06:54:45,852 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 06:54:46,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:46,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405746131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:46,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405746131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:46,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405746133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:46,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405746142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:46,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405746143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,196 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/20fed97db3094ed8b7912ec49ac73830 2023-04-25 06:54:46,205 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/e84b366603264f0d82fc7656650f4809 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/e84b366603264f0d82fc7656650f4809 2023-04-25 06:54:46,209 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/e84b366603264f0d82fc7656650f4809, entries=150, sequenceid=68, filesize=11.6 K 2023-04-25 06:54:46,210 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e52cbc39862f4846a76498bdde7095f7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e52cbc39862f4846a76498bdde7095f7 2023-04-25 06:54:46,214 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e52cbc39862f4846a76498bdde7095f7, entries=150, sequenceid=68, filesize=11.6 K 2023-04-25 06:54:46,215 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/20fed97db3094ed8b7912ec49ac73830 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/20fed97db3094ed8b7912ec49ac73830 2023-04-25 06:54:46,219 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/20fed97db3094ed8b7912ec49ac73830, entries=150, sequenceid=68, filesize=11.6 K 2023-04-25 06:54:46,220 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 11919fcb635bcaf9ea0c7782550d9d17 in 1257ms, sequenceid=68, compaction requested=false 2023-04-25 06:54:46,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:46,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:46,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:46,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:46,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:46,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebcf4f3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:46,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:46,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:46,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,228 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:46,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:46,232 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:46,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:46,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:46,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:46,232 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:46,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:46,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:46,235 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:46,237 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:46,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:46,239 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:46,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:46,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:46,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:46,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:46,240 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:46,240 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:46,340 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:46,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:46,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:46,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:46,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:46,344 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:46,344 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:46,344 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:46,344 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:46,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:46,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,347 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:46,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:46,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:46,348 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:46,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:46,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:46,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:46,349 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:46,349 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:46,349 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:46,764 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/0aac9d48011241119b125c1e47047d57 2023-04-25 06:54:46,779 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/65bcd588638546b3b45271d60de56602 2023-04-25 06:54:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:47,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:47,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405747151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405747152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405747152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405747153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405747153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,223 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/9f3f9d3d88f24b0589e70c4b2961cc2c 2023-04-25 06:54:47,229 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/0aac9d48011241119b125c1e47047d57 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0aac9d48011241119b125c1e47047d57 2023-04-25 06:54:47,233 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0aac9d48011241119b125c1e47047d57, entries=150, sequenceid=86, filesize=11.6 K 2023-04-25 06:54:47,234 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/65bcd588638546b3b45271d60de56602 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/65bcd588638546b3b45271d60de56602 2023-04-25 06:54:47,240 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/65bcd588638546b3b45271d60de56602, entries=150, sequenceid=86, filesize=11.6 K 2023-04-25 06:54:47,241 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/9f3f9d3d88f24b0589e70c4b2961cc2c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/9f3f9d3d88f24b0589e70c4b2961cc2c 2023-04-25 06:54:47,245 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/9f3f9d3d88f24b0589e70c4b2961cc2c, entries=150, sequenceid=86, filesize=11.6 K 2023-04-25 06:54:47,246 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 11919fcb635bcaf9ea0c7782550d9d17 in 897ms, sequenceid=86, compaction requested=true 2023-04-25 06:54:47,247 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:47,247 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:47,250 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,250 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6532c8f1[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:47,252 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:47,252 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,253 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:47,254 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,254 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:47,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:47,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:54:47,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:47,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:47,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:47,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:47,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:47,259 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:47,260 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:47,260 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:47,260 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:47,260 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:47,264 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,265 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,265 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,267 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405747268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405747269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405747270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405747270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:47,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:47,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:47,275 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:47,275 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:47,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405747273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,376 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:47,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405747374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405747374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405747375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405747375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:47,379 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:47,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405747377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:47,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:47,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,384 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:47,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:47,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:47,386 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,386 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:47,386 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:47,386 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:47,393 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:47,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c03505[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:47,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:47,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,396 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:47,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:47,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:47,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:47,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:47,398 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:47,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:47,400 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:47,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:47,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:47,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:47,400 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:47,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,403 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:47,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:47,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:47,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,410 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:47,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,410 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:47,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,510 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:47,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:47,514 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:47,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:47,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:47,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,516 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:47,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:47,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:47,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:47,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:47,518 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:47,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7722b8de[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:47,521 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:47,521 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,522 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:47,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:47,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:47,528 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:47,528 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:47,528 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:47,528 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:47,529 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:47,530 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,530 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,530 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:47,534 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:47,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,534 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:47,534 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:47,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405747578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405747579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405747579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405747580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405747582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,635 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:47,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:47,638 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:47,638 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,638 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:47,638 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:47,638 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,638 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,639 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:47,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:47,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:47,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:47,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:47,642 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,642 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:47,642 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:47,642 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:47,645 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e00c1a5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:47,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:47,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,648 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:47,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:47,650 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:47,653 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:47,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:47,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:47,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:47,653 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:47,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,659 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:47,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,660 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:47,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:47,660 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:47,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/ca7066f7b9dc436fa931432b8a5acbdf 2023-04-25 06:54:47,760 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:47,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:47,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:47,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:47,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:47,763 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:47,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:47,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:47,764 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,764 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,765 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:47,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:47,767 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,767 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:47,767 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:47,767 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:47,768 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,768 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,769 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,769 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d06fdd4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:47,770 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:47,770 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,772 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:47,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:47,773 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:47,773 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:47,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:47,776 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:47,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:47,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:47,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:47,776 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:47,778 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,778 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,778 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,782 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,782 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:47,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:47,782 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,782 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,783 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:47,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:47,782 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,783 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:47,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,883 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:47,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:47,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405747882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:47,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405747883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405747883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405747884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:47,886 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:47,886 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,886 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:47,886 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:47,887 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,887 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,889 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,889 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,889 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,889 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:47,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:47,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405747888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:47,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:47,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:47,890 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:47,890 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:47,890 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:47,891 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,891 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:47,891 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:47,891 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:47,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,892 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,893 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,893 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:47,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1e8f6c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:47,894 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:47,894 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,895 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:47,897 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,897 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:47,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:47,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:47,901 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:47,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:47,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:47,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:47,901 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:47,904 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,904 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,904 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:47,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:47,905 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:47,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:47,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:47,907 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:47,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:47,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:47,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:47,908 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:47,908 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:47,908 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,908 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:47,908 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:47,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:47,908 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:47,908 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,008 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:48,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:48,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:48,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:48,012 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:48,012 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,012 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:48,012 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:48,012 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,012 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,014 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:48,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,016 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:48,016 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:48,016 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:48,018 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,018 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,018 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:48,019 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f982a59[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:48,019 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:48,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:48,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:48,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:48,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:48,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:48,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:48,026 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:48,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:48,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:48,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:48,027 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:48,028 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,028 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,028 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:48,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:48,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,033 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,033 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,033 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:48,033 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,033 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,033 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,033 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/ff7a6528957c4e7baa9ebcbcdf6d9e13 2023-04-25 06:54:48,133 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:48,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:48,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:48,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:48,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:48,136 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:48,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:48,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:48,137 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,137 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,138 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,139 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,139 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:48,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:48,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:48,140 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,140 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:48,141 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:48,141 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:48,142 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,143 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,143 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,143 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:48,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ddcbe6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:48,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:48,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,145 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:48,145 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:48,149 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,149 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:48,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:48,152 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:48,152 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:48,152 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:48,152 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:48,152 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:48,154 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,154 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,154 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,154 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:48,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,158 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:48,158 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,259 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:48,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:48,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:48,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:48,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:48,262 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:48,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:48,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:48,263 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,263 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,264 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:48,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:48,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:48,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:48,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:48,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:48,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:48,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:48,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:48,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:48,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3f7294[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:48,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:48,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:48,278 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,278 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:48,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:48,279 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:48,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:48,282 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:48,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:48,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:48,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:48,282 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:48,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:48,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:48,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,289 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:48,289 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:48,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405748387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,390 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:48,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:48,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:48,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405748389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:48,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405748389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:48,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:48,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51280 deadline: 1682405748390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:48,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:48,393 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:48,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:48,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:48,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,395 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:48,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405748394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:48,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:48,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:48,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:48,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:48,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:48,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:48,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:48,399 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:48,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e3f5b0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:48,401 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:48,401 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:48,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:48,405 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:48,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,405 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,405 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:48,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:48,407 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:48,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:48,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:48,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:48,407 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:48,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,429 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,429 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,429 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:48,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:48,429 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,429 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,430 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:48,430 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/6c38bfcee0ce42a78cf2f7623a40f773 2023-04-25 06:54:48,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/ca7066f7b9dc436fa931432b8a5acbdf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ca7066f7b9dc436fa931432b8a5acbdf 2023-04-25 06:54:48,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ca7066f7b9dc436fa931432b8a5acbdf, entries=150, sequenceid=104, filesize=11.6 K 2023-04-25 06:54:48,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/ff7a6528957c4e7baa9ebcbcdf6d9e13 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ff7a6528957c4e7baa9ebcbcdf6d9e13 2023-04-25 06:54:48,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ff7a6528957c4e7baa9ebcbcdf6d9e13, entries=150, sequenceid=104, filesize=11.6 K 2023-04-25 06:54:48,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/6c38bfcee0ce42a78cf2f7623a40f773 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6c38bfcee0ce42a78cf2f7623a40f773 2023-04-25 06:54:48,530 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:48,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:48,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:48,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:48,535 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:48,535 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,535 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:48,535 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:48,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6c38bfcee0ce42a78cf2f7623a40f773, entries=150, sequenceid=104, filesize=11.6 K 2023-04-25 06:54:48,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 11919fcb635bcaf9ea0c7782550d9d17 in 1278ms, sequenceid=104, compaction requested=true 2023-04-25 06:54:48,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:48,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:48,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:48,536 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:48,536 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:48,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:48,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,538 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:48,538 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:48,538 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/A is initiating minor compaction (all files) 2023-04-25 06:54:48,538 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/B is initiating minor compaction (all files) 2023-04-25 06:54:48,538 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/B in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,538 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/A in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,538 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c16d57075564c60989b4c5afff20dce, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e52cbc39862f4846a76498bdde7095f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/65bcd588638546b3b45271d60de56602, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ff7a6528957c4e7baa9ebcbcdf6d9e13] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=46.5 K 2023-04-25 06:54:48,538 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c72b299468af4d0eb466fe4143e07293, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/e84b366603264f0d82fc7656650f4809, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0aac9d48011241119b125c1e47047d57, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ca7066f7b9dc436fa931432b8a5acbdf] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=46.5 K 2023-04-25 06:54:48,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,538 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3c16d57075564c60989b4c5afff20dce, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405684061 2023-04-25 06:54:48,538 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c72b299468af4d0eb466fe4143e07293, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405684061 2023-04-25 06:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,539 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e52cbc39862f4846a76498bdde7095f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682405684391 2023-04-25 06:54:48,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:48,539 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting e84b366603264f0d82fc7656650f4809, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682405684391 2023-04-25 06:54:48,539 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 65bcd588638546b3b45271d60de56602, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682405685006 2023-04-25 06:54:48,539 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 0aac9d48011241119b125c1e47047d57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682405685006 2023-04-25 06:54:48,539 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ff7a6528957c4e7baa9ebcbcdf6d9e13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682405687151 2023-04-25 06:54:48,539 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting ca7066f7b9dc436fa931432b8a5acbdf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682405687151 2023-04-25 06:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:48,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:48,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:48,561 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#B#compaction#211 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:48,562 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#A#compaction#210 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:48,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:48,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:48,562 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,562 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:48,562 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:48,596 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/655f6efb836845cc9655f1836e664f42 2023-04-25 06:54:48,609 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e312277148b24662bdd9d9bb2e11b8ab 2023-04-25 06:54:48,621 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/0c43f8dfee0146178a96d5e3130226b4 2023-04-25 06:54:48,626 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/655f6efb836845cc9655f1836e664f42 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/655f6efb836845cc9655f1836e664f42 2023-04-25 06:54:48,632 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/655f6efb836845cc9655f1836e664f42, entries=150, sequenceid=118, filesize=11.6 K 2023-04-25 06:54:48,633 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e312277148b24662bdd9d9bb2e11b8ab as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e312277148b24662bdd9d9bb2e11b8ab 2023-04-25 06:54:48,637 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e312277148b24662bdd9d9bb2e11b8ab, entries=150, sequenceid=118, filesize=11.6 K 2023-04-25 06:54:48,638 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/0c43f8dfee0146178a96d5e3130226b4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/0c43f8dfee0146178a96d5e3130226b4 2023-04-25 06:54:48,644 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/0c43f8dfee0146178a96d5e3130226b4, entries=150, sequenceid=118, filesize=11.6 K 2023-04-25 06:54:48,645 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 11919fcb635bcaf9ea0c7782550d9d17 in 83ms, sequenceid=118, compaction requested=true 2023-04-25 06:54:48,645 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:48,645 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:48,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:48,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:48,649 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ac7c797[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:48,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:48,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,655 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,655 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:48,655 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:48,655 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:48,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:48,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:48,660 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:48,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:48,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:48,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:48,660 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:48,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:48,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,672 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:48,672 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,772 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:48,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:48,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:48,776 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:48,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:48,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:48,777 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,777 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,778 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,778 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:48,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:48,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:48,780 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,780 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:48,782 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,783 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,783 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33ccb9d2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:48,785 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:48,785 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:48,786 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,786 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,789 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:48,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:48,791 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:48,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:48,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:48,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:48,791 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:48,793 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,793 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,793 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,797 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,797 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,797 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,797 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:48,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:48,797 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:48,798 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:48,798 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,797 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,898 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:48,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:48,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:48,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:48,902 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:48,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:48,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:48,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,904 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,905 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,905 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:48,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:48,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:48,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:48,906 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,906 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:48,906 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:48,906 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:48,909 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:48,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f1d6ed[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:48,911 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:48,911 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,912 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,912 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:48,912 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:48,913 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:48,913 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:48,914 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:48,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:48,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:48,917 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:48,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:48,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:48,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:48,918 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:48,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:48,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:48,921 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:48,924 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:48,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:48,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:48,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:48,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,925 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:48,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:48,925 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:48,925 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:48,925 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:48,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:48,986 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/7d026c3eccc8445a820314e191e981f6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7d026c3eccc8445a820314e191e981f6 2023-04-25 06:54:48,990 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/B of 11919fcb635bcaf9ea0c7782550d9d17 into 7d026c3eccc8445a820314e191e981f6(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:48,990 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:48,990 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/B, priority=12, startTime=1682405688536; duration=0sec 2023-04-25 06:54:48,991 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:48,991 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:48,992 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59533 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:48,993 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/C is initiating minor compaction (all files) 2023-04-25 06:54:48,993 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/C in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:48,993 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/bc10b4c54e8a4627a5e2321ec36e6b66, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/20fed97db3094ed8b7912ec49ac73830, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/9f3f9d3d88f24b0589e70c4b2961cc2c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6c38bfcee0ce42a78cf2f7623a40f773, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/0c43f8dfee0146178a96d5e3130226b4] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=58.1 K 2023-04-25 06:54:48,993 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting bc10b4c54e8a4627a5e2321ec36e6b66, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682405684061 2023-04-25 06:54:48,994 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 20fed97db3094ed8b7912ec49ac73830, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682405684391 2023-04-25 06:54:48,994 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9f3f9d3d88f24b0589e70c4b2961cc2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682405685006 2023-04-25 06:54:48,994 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6c38bfcee0ce42a78cf2f7623a40f773, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682405687151 2023-04-25 06:54:48,995 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 0c43f8dfee0146178a96d5e3130226b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682405687267 2023-04-25 06:54:48,999 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/3b5422bf3e1c436183391160f6f57a1a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b5422bf3e1c436183391160f6f57a1a 2023-04-25 06:54:49,005 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#C#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:49,005 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/A of 11919fcb635bcaf9ea0c7782550d9d17 into 3b5422bf3e1c436183391160f6f57a1a(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:49,005 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:49,005 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/A, priority=12, startTime=1682405688536; duration=0sec 2023-04-25 06:54:49,005 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:49,021 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/26f685b5274f4239a05c523d280d9771 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26f685b5274f4239a05c523d280d9771 2023-04-25 06:54:49,025 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:49,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:49,026 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/C of 11919fcb635bcaf9ea0c7782550d9d17 into 26f685b5274f4239a05c523d280d9771(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:49,026 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:49,026 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/C, priority=11, startTime=1682405688536; duration=0sec 2023-04-25 06:54:49,027 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:49,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:49,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:49,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:49,029 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:49,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:49,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:49,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:49,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:49,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:49,035 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,035 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:49,035 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:49,035 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:49,035 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:49,037 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:49,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e74e996[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:49,039 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:49,039 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,040 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:49,040 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,040 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:49,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,042 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,042 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:49,045 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:49,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:49,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:49,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:49,045 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:49,047 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,047 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,047 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,048 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,053 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:49,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:49,054 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,054 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,054 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,054 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:49,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,054 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:49,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:49,154 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:49,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:49,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:49,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:49,158 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:49,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:49,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:49,159 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,159 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,161 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:49,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:49,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:49,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:49,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:49,163 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:49,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,165 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,165 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,165 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:49,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8b70b0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:49,167 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:49,167 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,168 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,169 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:49,169 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:49,169 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:49,169 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,169 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:49,170 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:49,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:49,173 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:49,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:49,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:49,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:49,173 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:49,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,176 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,180 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:49,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:49,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,181 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:49,181 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:49,180 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:49,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,281 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:49,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:49,285 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:49,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:49,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:49,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,287 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,287 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:49,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:49,288 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,288 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:49,289 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:49,289 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:49,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,291 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,291 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:49,292 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c5f461a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:49,292 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:49,295 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,295 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:49,297 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:49,297 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,297 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:49,297 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:49,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:49,300 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:49,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:49,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:49,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:49,300 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:49,302 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,302 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,302 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,303 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,304 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:49,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,306 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:49,307 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:49,407 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:49,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:49,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:49,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:49,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:49,409 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:49,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:49,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:49,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:49,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:49,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:49,416 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,416 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:49,420 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-25 06:54:49,421 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:49,421 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:49,421 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:49,422 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:49,422 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:49,422 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:49,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:49,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:49,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405689439, max timestamp=1682405689442Num uniques 859; 2023-04-25 06:54:49,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:49,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405689439, max timestamp=1682405689442Num uniques 859; 2023-04-25 06:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405689439, max timestamp=1682405689442Num uniques 859; 2023-04-25 06:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:49,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51278 deadline: 1682405749456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682405689449, max timestamp=1682405689456Num uniques 1068; 2023-04-25 06:54:49,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-25 06:54:49,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682405689453, max timestamp=1682405689456Num uniques 1068; 2023-04-25 06:54:49,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682405689453, max timestamp=1682405689456Num uniques 1068; 2023-04-25 06:54:49,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-25 06:54:49,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-25 06:54:49,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682405689466, max timestamp=1682405689475Num uniques 1009; 2023-04-25 06:54:49,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682405689466, max timestamp=1682405689475Num uniques 1127; 2023-04-25 06:54:49,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682405689466, max timestamp=1682405689475Num uniques 1009; 2023-04-25 06:54:49,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-25 06:54:49,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12682, suffix heap size=132904,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=277 2023-04-25 06:54:49,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12682, suffix heap size=132904,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=277 2023-04-25 06:54:49,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405749494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51348 deadline: 1682405749495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682405689490, max timestamp=1682405689496Num uniques 1136; 2023-04-25 06:54:49,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-25 06:54:49,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682405689490, max timestamp=1682405689496Num uniques 1136; 2023-04-25 06:54:49,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682405689490, max timestamp=1682405689496Num uniques 1136; 2023-04-25 06:54:49,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-25 06:54:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51322 deadline: 1682405749496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-25 06:54:49,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682405689542, max timestamp=1682405689555Num uniques 1145; 2023-04-25 06:54:49,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-25 06:54:49,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682405689542, max timestamp=1682405689555Num uniques 1145; 2023-04-25 06:54:49,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-25 06:54:49,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682405689542, max timestamp=1682405689555Num uniques 1145; 2023-04-25 06:54:49,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-25 06:54:49,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682405689575, max timestamp=1682405689590Num uniques 1054; 2023-04-25 06:54:49,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682405689575, max timestamp=1682405689590Num uniques 1054; 2023-04-25 06:54:49,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 06:54:49,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 06:54:49,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682405689575, max timestamp=1682405689590Num uniques 1054; 2023-04-25 06:54:49,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 06:54:49,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405749636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682405689596, max timestamp=1682405689639Num uniques 1063; 2023-04-25 06:54:49,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-25 06:54:49,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682405689596, max timestamp=1682405689639Num uniques 1063; 2023-04-25 06:54:49,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682405689596, max timestamp=1682405689639Num uniques 1063; 2023-04-25 06:54:49,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-25 06:54:49,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-25 06:54:49,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682405689653, max timestamp=1682405689659Num uniques 1072; 2023-04-25 06:54:49,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682405689653, max timestamp=1682405689659Num uniques 1072; 2023-04-25 06:54:49,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-25 06:54:49,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-25 06:54:49,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682405689653, max timestamp=1682405689659Num uniques 1072; 2023-04-25 06:54:49,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-25 06:54:49,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682405689673, max timestamp=1682405689678Num uniques 1081; 2023-04-25 06:54:49,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682405689673, max timestamp=1682405689678Num uniques 1081; 2023-04-25 06:54:49,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 06:54:49,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682405689673, max timestamp=1682405689678Num uniques 1081; 2023-04-25 06:54:49,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 06:54:49,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 06:54:49,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682405689694, max timestamp=1682405689700Num uniques 1040; 2023-04-25 06:54:49,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-25 06:54:49,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682405689694, max timestamp=1682405689700Num uniques 1040; 2023-04-25 06:54:49,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682405689694, max timestamp=1682405689700Num uniques 1040; 2023-04-25 06:54:49,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-25 06:54:49,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-25 06:54:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1682405689715, max timestamp=1682405689719Num uniques 1149; 2023-04-25 06:54:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405749719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52614, new segment data size=11394, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-25 06:54:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1682405689715, max timestamp=1682405689719Num uniques 1149; 2023-04-25 06:54:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52614, new segment data size=11394, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-25 06:54:49,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1682405689715, max timestamp=1682405689719Num uniques 1149; 2023-04-25 06:54:49,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52614, new segment data size=11394, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-25 06:54:49,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405749742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682405689734, max timestamp=1682405689744Num uniques 1108; 2023-04-25 06:54:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-25 06:54:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682405689734, max timestamp=1682405689744Num uniques 1108; 2023-04-25 06:54:49,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-25 06:54:49,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682405689734, max timestamp=1682405689744Num uniques 1108; 2023-04-25 06:54:49,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-25 06:54:49,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682405689785, max timestamp=1682405689791Num uniques 1017; 2023-04-25 06:54:49,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-25 06:54:49,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682405689785, max timestamp=1682405689791Num uniques 1017; 2023-04-25 06:54:49,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682405689785, max timestamp=1682405689791Num uniques 1017; 2023-04-25 06:54:49,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-25 06:54:49,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-25 06:54:49,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682405689809, max timestamp=1682405689814Num uniques 1026; 2023-04-25 06:54:49,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-25 06:54:49,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682405689809, max timestamp=1682405689814Num uniques 1026; 2023-04-25 06:54:49,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-25 06:54:49,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682405689809, max timestamp=1682405689814Num uniques 1026; 2023-04-25 06:54:49,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-25 06:54:49,830 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/227e012bb18540cdad88dcc6c0181879 2023-04-25 06:54:49,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682405689832, max timestamp=1682405689838Num uniques 1035; 2023-04-25 06:54:49,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-25 06:54:49,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682405689832, max timestamp=1682405689838Num uniques 1035; 2023-04-25 06:54:49,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682405689832, max timestamp=1682405689838Num uniques 1035; 2023-04-25 06:54:49,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-25 06:54:49,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-25 06:54:49,843 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/b87a1cab8a8e4495af7c9a33963e5cd6 2023-04-25 06:54:49,853 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/c0c484dfa26d4fd5bdf23c9c98203705 2023-04-25 06:54:49,857 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/227e012bb18540cdad88dcc6c0181879 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/227e012bb18540cdad88dcc6c0181879 2023-04-25 06:54:49,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11164, totalHeapSize=25616, min timestamp=1682405689849, max timestamp=1682405689860Num uniques 1044; 2023-04-25 06:54:49,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47804, new segment data size=11164, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-25 06:54:49,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11164, totalHeapSize=25616, min timestamp=1682405689849, max timestamp=1682405689860Num uniques 1044; 2023-04-25 06:54:49,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11168, totalHeapSize=25616, min timestamp=1682405689849, max timestamp=1682405689860Num uniques 1044; 2023-04-25 06:54:49,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47804, new segment data size=11164, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-25 06:54:49,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47808, new segment data size=11168, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-25 06:54:49,873 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/227e012bb18540cdad88dcc6c0181879, entries=100, sequenceid=133, filesize=9.4 K 2023-04-25 06:54:49,874 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/b87a1cab8a8e4495af7c9a33963e5cd6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/b87a1cab8a8e4495af7c9a33963e5cd6 2023-04-25 06:54:49,882 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/b87a1cab8a8e4495af7c9a33963e5cd6, entries=100, sequenceid=133, filesize=9.4 K 2023-04-25 06:54:49,883 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/c0c484dfa26d4fd5bdf23c9c98203705 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/c0c484dfa26d4fd5bdf23c9c98203705 2023-04-25 06:54:49,888 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/c0c484dfa26d4fd5bdf23c9c98203705, entries=100, sequenceid=133, filesize=9.4 K 2023-04-25 06:54:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682405689873, max timestamp=1682405689887Num uniques 1103; 2023-04-25 06:54:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-25 06:54:49,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682405689873, max timestamp=1682405689887Num uniques 1103; 2023-04-25 06:54:49,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-25 06:54:49,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682405689873, max timestamp=1682405689887Num uniques 1103; 2023-04-25 06:54:49,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-25 06:54:49,890 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=40.25 KB/41220 for 11919fcb635bcaf9ea0c7782550d9d17 in 474ms, sequenceid=133, compaction requested=true 2023-04-25 06:54:49,891 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:49,891 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,893 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:49,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=53.67 KB heapSize=137.11 KB 2023-04-25 06:54:49,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:49,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:49,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:49,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:49,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:49,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:49,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b2b9c2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:49,896 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:49,896 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,897 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:49,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:49,899 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:49,904 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:49,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:49,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:49,904 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:49,904 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:49,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:49,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,907 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,908 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:49,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:49,911 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:49,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:49,911 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:49,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:49,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:49,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:49,911 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:49,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682405689913, max timestamp=1682405689925Num uniques 859; 2023-04-25 06:54:49,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682405689913, max timestamp=1682405689925Num uniques 859; 2023-04-25 06:54:49,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:49,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:49,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682405689913, max timestamp=1682405689925Num uniques 859; 2023-04-25 06:54:49,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:49,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405749956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405749957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405749958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405749958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:49,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:49,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405749959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,012 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:50,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:50,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:50,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:50,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:50,015 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:50,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:50,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:50,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,016 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,019 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,019 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,020 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,020 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:50,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:50,021 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:50,024 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec3cafb[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:50,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:50,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,027 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:50,029 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:50,031 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:50,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:50,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:50,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:50,031 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:50,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,033 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,034 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:50,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:50,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,042 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,042 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:50,042 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405750060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405750061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405750061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405750062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405750063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,142 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:50,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:50,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:50,146 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:50,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:50,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:50,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:50,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:50,149 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:50,152 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:50,153 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:50,153 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7c2f63[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:50,154 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:50,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:50,156 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:50,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:50,159 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:50,159 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:50,159 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:50,159 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:50,159 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:50,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,161 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,162 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,164 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:50,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:50,164 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,164 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:50,165 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:50,165 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:50,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,265 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:50,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:50,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405750263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405750264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405750264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405750266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:50,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405750266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:50,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:50,269 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:50,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:50,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:50,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,270 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:50,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:50,272 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,272 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:50,273 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:50,273 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:50,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d5db00[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:50,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:50,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:50,278 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,278 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:50,279 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:50,282 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:50,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:50,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:50,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:50,282 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:50,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:50,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:50,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,291 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,291 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:50,291 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:50,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/0591aac787b044a79386300f2ac06ffb 2023-04-25 06:54:50,392 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:50,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:50,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:50,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:50,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:50,396 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:50,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:50,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:50,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,397 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,398 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:50,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:50,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:50,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:50,404 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:50,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bab1b51[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:50,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:50,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,408 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:50,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:50,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:50,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:50,412 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:50,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:50,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:50,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:50,412 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:50,413 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,413 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,413 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,414 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,416 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,418 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,418 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:50,418 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:50,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,518 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:50,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:50,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:50,523 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:50,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:50,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:50,524 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,524 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,526 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:50,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:50,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:50,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:50,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:50,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:50,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:50,528 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,528 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:50,529 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:50,529 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:50,530 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:50,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bff88cd[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:50,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:50,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,533 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,533 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,533 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:50,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:50,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:50,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:50,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:50,535 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:50,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:50,537 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:50,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:50,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:50,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:50,537 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:50,539 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,539 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,540 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,544 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:50,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:50,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,544 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:50,544 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:50,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,545 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,545 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405750568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405750568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405750568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405750570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405750571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,645 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:50,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:50,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:50,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:50,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:50,648 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:50,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:50,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:50,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,650 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,650 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,650 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,650 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:50,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:50,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:50,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:50,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:50,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:50,652 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,652 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:50,653 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:50,653 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,653 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:50,653 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:50,653 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,653 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:50,653 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6576b5e6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:50,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:50,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,657 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:50,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:50,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:50,658 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,658 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:50,660 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:50,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:50,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:50,662 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:50,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:50,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:50,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:50,662 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:50,664 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,664 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:50,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,664 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:50,665 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:50,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:50,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:50,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:50,668 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:50,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:50,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,669 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:50,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:50,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:50,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/21600154ecba45dcbc48fb4832f13194 2023-04-25 06:54:50,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/67b0ef7b9c27446f913376f2f0796f52 2023-04-25 06:54:50,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/0591aac787b044a79386300f2ac06ffb as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0591aac787b044a79386300f2ac06ffb 2023-04-25 06:54:50,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0591aac787b044a79386300f2ac06ffb, entries=150, sequenceid=431, filesize=11.9 K 2023-04-25 06:54:50,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/21600154ecba45dcbc48fb4832f13194 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/21600154ecba45dcbc48fb4832f13194 2023-04-25 06:54:50,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/21600154ecba45dcbc48fb4832f13194, entries=150, sequenceid=431, filesize=11.9 K 2023-04-25 06:54:50,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/67b0ef7b9c27446f913376f2f0796f52 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/67b0ef7b9c27446f913376f2f0796f52 2023-04-25 06:54:50,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/67b0ef7b9c27446f913376f2f0796f52, entries=150, sequenceid=431, filesize=11.9 K 2023-04-25 06:54:50,766 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~137.06 KB/140352, currentSize=134.18 KB/137400 for 11919fcb635bcaf9ea0c7782550d9d17 in 873ms, sequenceid=431, compaction requested=true 2023-04-25 06:54:50,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:50,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:50,766 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:50,766 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:54:50,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:50,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:50,767 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45840 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:50,767 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/A is initiating minor compaction (all files) 2023-04-25 06:54:50,767 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45840 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:54:50,767 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/A in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/B is initiating minor compaction (all files) 2023-04-25 06:54:50,768 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b5422bf3e1c436183391160f6f57a1a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/655f6efb836845cc9655f1836e664f42, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/227e012bb18540cdad88dcc6c0181879, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0591aac787b044a79386300f2ac06ffb] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=44.8 K 2023-04-25 06:54:50,768 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/B in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,768 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7d026c3eccc8445a820314e191e981f6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e312277148b24662bdd9d9bb2e11b8ab, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/b87a1cab8a8e4495af7c9a33963e5cd6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/21600154ecba45dcbc48fb4832f13194] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=44.8 K 2023-04-25 06:54:50,768 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 3b5422bf3e1c436183391160f6f57a1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682405687151 2023-04-25 06:54:50,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7d026c3eccc8445a820314e191e981f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1682405687151 2023-04-25 06:54:50,769 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 655f6efb836845cc9655f1836e664f42, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682405687267 2023-04-25 06:54:50,769 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e312277148b24662bdd9d9bb2e11b8ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682405687267 2023-04-25 06:54:50,769 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:50,769 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 227e012bb18540cdad88dcc6c0181879, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682405689396 2023-04-25 06:54:50,770 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b87a1cab8a8e4495af7c9a33963e5cd6, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682405689396 2023-04-25 06:54:50,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:50,770 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 0591aac787b044a79386300f2ac06ffb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682405689880 2023-04-25 06:54:50,770 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 21600154ecba45dcbc48fb4832f13194, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682405689880 2023-04-25 06:54:50,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:50,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:50,779 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:50,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:50,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:50,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,779 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,781 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:50,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:50,781 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:50,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:50,785 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#B#compaction#222 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:50,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:50,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:50,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:50,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:50,801 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,801 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:50,801 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#A#compaction#223 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:50,802 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-25 06:54:50,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:50,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:50,803 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:50,803 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:50,803 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:50,803 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:50,817 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/4bb77516f48c41768659a06ba04921c6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bb77516f48c41768659a06ba04921c6 2023-04-25 06:54:50,829 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/B of 11919fcb635bcaf9ea0c7782550d9d17 into 4bb77516f48c41768659a06ba04921c6(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:50,829 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:50,829 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/B, priority=12, startTime=1682405690766; duration=0sec 2023-04-25 06:54:50,829 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:50,829 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:54:50,831 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:54:50,831 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/C is initiating minor compaction (all files) 2023-04-25 06:54:50,831 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/C in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:50,831 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26f685b5274f4239a05c523d280d9771, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/c0c484dfa26d4fd5bdf23c9c98203705, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/67b0ef7b9c27446f913376f2f0796f52] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=33.2 K 2023-04-25 06:54:50,832 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 26f685b5274f4239a05c523d280d9771, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682405687267 2023-04-25 06:54:50,834 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c0c484dfa26d4fd5bdf23c9c98203705, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682405689396 2023-04-25 06:54:50,835 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 67b0ef7b9c27446f913376f2f0796f52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682405689880 2023-04-25 06:54:50,842 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#C#compaction#225 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:50,852 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/38825da17c0840698f6c5ca1db354774 2023-04-25 06:54:50,874 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/ae1e6c3fb3e24c4d8ebc3354f49f45d3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/ae1e6c3fb3e24c4d8ebc3354f49f45d3 2023-04-25 06:54:50,879 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/C of 11919fcb635bcaf9ea0c7782550d9d17 into ae1e6c3fb3e24c4d8ebc3354f49f45d3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:50,879 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:50,879 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/C, priority=13, startTime=1682405690766; duration=0sec 2023-04-25 06:54:50,879 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:51,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:51,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:51,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405751078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405751079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405751079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405751079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405751080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405751183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405751184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405751184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405751184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405751184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,220 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/54702c21b2314da3b8e64e56d3d1bcd8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/54702c21b2314da3b8e64e56d3d1bcd8 2023-04-25 06:54:51,224 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/A of 11919fcb635bcaf9ea0c7782550d9d17 into 54702c21b2314da3b8e64e56d3d1bcd8(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:51,224 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:51,224 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/A, priority=12, startTime=1682405690766; duration=0sec 2023-04-25 06:54:51,224 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:51,281 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/ef037edda9094a06abfe8cdebc6fc818 2023-04-25 06:54:51,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405751386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405751387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405751387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405751387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405751387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,692 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/648084c7372949758c8159a83dd54d12 2023-04-25 06:54:51,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405751689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405751689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405751689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405751691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:51,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405751692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,696 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/38825da17c0840698f6c5ca1db354774 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/38825da17c0840698f6c5ca1db354774 2023-04-25 06:54:51,700 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/38825da17c0840698f6c5ca1db354774, entries=150, sequenceid=467, filesize=11.9 K 2023-04-25 06:54:51,701 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/ef037edda9094a06abfe8cdebc6fc818 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ef037edda9094a06abfe8cdebc6fc818 2023-04-25 06:54:51,705 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ef037edda9094a06abfe8cdebc6fc818, entries=150, sequenceid=467, filesize=11.9 K 2023-04-25 06:54:51,706 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/648084c7372949758c8159a83dd54d12 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/648084c7372949758c8159a83dd54d12 2023-04-25 06:54:51,709 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/648084c7372949758c8159a83dd54d12, entries=150, sequenceid=467, filesize=11.9 K 2023-04-25 06:54:51,710 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 11919fcb635bcaf9ea0c7782550d9d17 in 908ms, sequenceid=467, compaction requested=false 2023-04-25 06:54:51,710 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:51,710 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:51,718 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627c7f93[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:51,720 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:51,720 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,721 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,721 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:51,721 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:51,721 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:51,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:51,725 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,725 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:51,725 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:51,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:51,728 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:51,728 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:51,728 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:51,728 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:51,728 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:51,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,731 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:51,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:51,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:51,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:51,733 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:51,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:51,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:51,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:51,735 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:51,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:51,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:51,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:51,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,736 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:51,736 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:51,836 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:51,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:51,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:51,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:51,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:51,839 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:51,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:51,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:51,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:51,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,840 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:51,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:51,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:51,841 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:51,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:51,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:51,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:51,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:51,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:51,843 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:51,843 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:51,843 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:54:51,843 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:51,843 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:51,844 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:51,844 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:51,844 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:51,844 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:52,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:52,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:52,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405692208, max timestamp=1682405692212Num uniques 859; 2023-04-25 06:54:52,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:52,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405692208, max timestamp=1682405692212Num uniques 859; 2023-04-25 06:54:52,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:52,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405692208, max timestamp=1682405692212Num uniques 859; 2023-04-25 06:54:52,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:52,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405752224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405752224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405752227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405752228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405752228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,253 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/32605f0d975947e2a9d956145f1d4b11 2023-04-25 06:54:52,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405752329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405752329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405752332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405752332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405752332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405752532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405752532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405752535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405752536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405752536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,664 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/217def7fef7a4618b8f0e8e546d194cb 2023-04-25 06:54:52,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405752834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405752835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405752838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405752838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:52,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:52,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405752840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,076 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/b593fe7d6e1149d2813189817b576a8d 2023-04-25 06:54:53,080 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/32605f0d975947e2a9d956145f1d4b11 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/32605f0d975947e2a9d956145f1d4b11 2023-04-25 06:54:53,085 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/32605f0d975947e2a9d956145f1d4b11, entries=150, sequenceid=480, filesize=11.9 K 2023-04-25 06:54:53,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/217def7fef7a4618b8f0e8e546d194cb as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/217def7fef7a4618b8f0e8e546d194cb 2023-04-25 06:54:53,090 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/217def7fef7a4618b8f0e8e546d194cb, entries=150, sequenceid=480, filesize=11.9 K 2023-04-25 06:54:53,091 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/b593fe7d6e1149d2813189817b576a8d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/b593fe7d6e1149d2813189817b576a8d 2023-04-25 06:54:53,094 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/b593fe7d6e1149d2813189817b576a8d, entries=150, sequenceid=480, filesize=11.9 K 2023-04-25 06:54:53,095 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 11919fcb635bcaf9ea0c7782550d9d17 in 1252ms, sequenceid=480, compaction requested=true 2023-04-25 06:54:53,095 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:53,095 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fe811e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:53,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:53,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,101 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:53,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:53,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:53,103 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:53,103 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:53,105 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:53,105 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:53,105 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:53,105 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:53,105 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:53,107 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:53,107 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,107 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:53,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:53,108 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:53,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:53,114 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:53,114 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:53,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:53,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:53,115 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:53,115 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:53,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:53,215 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:53,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:53,218 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:53,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:53,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:53,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:53,221 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,221 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:53,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:53,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:53,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:53,223 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:53,223 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:53,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:53,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:53,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405753345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405753348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405753348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405753349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405753350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405753451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405753452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405753454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405753454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,633 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/10be9d8f7b1d4f6fba0462a15e3a2fcf 2023-04-25 06:54:53,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405753653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405753654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405753656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405753656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405753958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405753958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51342 deadline: 1682405753961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:53,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:53,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405753961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,079 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/a66feae88d004d9f91fa1ec990c97a9a 2023-04-25 06:54:54,100 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/44cf1afc093e4cb29ea5f130d827a54f 2023-04-25 06:54:54,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/10be9d8f7b1d4f6fba0462a15e3a2fcf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/10be9d8f7b1d4f6fba0462a15e3a2fcf 2023-04-25 06:54:54,110 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/10be9d8f7b1d4f6fba0462a15e3a2fcf, entries=150, sequenceid=517, filesize=11.9 K 2023-04-25 06:54:54,111 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/a66feae88d004d9f91fa1ec990c97a9a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/a66feae88d004d9f91fa1ec990c97a9a 2023-04-25 06:54:54,120 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/a66feae88d004d9f91fa1ec990c97a9a, entries=150, sequenceid=517, filesize=11.9 K 2023-04-25 06:54:54,121 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/44cf1afc093e4cb29ea5f130d827a54f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/44cf1afc093e4cb29ea5f130d827a54f 2023-04-25 06:54:54,125 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/44cf1afc093e4cb29ea5f130d827a54f, entries=150, sequenceid=517, filesize=11.9 K 2023-04-25 06:54:54,126 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 11919fcb635bcaf9ea0c7782550d9d17 in 903ms, sequenceid=517, compaction requested=true 2023-04-25 06:54:54,126 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:54,126 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:54,127 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d0da0c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:54,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:54,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,130 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:54,131 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,131 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:54,132 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:54,135 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:54,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:54,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:54,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:54,135 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:54,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,139 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,141 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:54,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,143 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,143 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:54,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,244 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:54,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:54,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:54,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:54,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:54,249 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:54,249 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,249 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:54,249 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:54,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:54,253 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:54,253 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:54,260 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/564f13e322d34aad8e32eacf4461d121 2023-04-25 06:54:54,272 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/3c84b267a78c459d9be7ef6c23ee90a4 2023-04-25 06:54:54,282 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/1dc4236735ba4c048545a7e75573abf5 2023-04-25 06:54:54,286 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/564f13e322d34aad8e32eacf4461d121 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/564f13e322d34aad8e32eacf4461d121 2023-04-25 06:54:54,291 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/564f13e322d34aad8e32eacf4461d121, entries=100, sequenceid=526, filesize=9.5 K 2023-04-25 06:54:54,292 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/3c84b267a78c459d9be7ef6c23ee90a4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c84b267a78c459d9be7ef6c23ee90a4 2023-04-25 06:54:54,296 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c84b267a78c459d9be7ef6c23ee90a4, entries=100, sequenceid=526, filesize=9.5 K 2023-04-25 06:54:54,297 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/1dc4236735ba4c048545a7e75573abf5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/1dc4236735ba4c048545a7e75573abf5 2023-04-25 06:54:54,302 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/1dc4236735ba4c048545a7e75573abf5, entries=100, sequenceid=526, filesize=9.5 K 2023-04-25 06:54:54,303 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 11919fcb635bcaf9ea0c7782550d9d17 in 50ms, sequenceid=526, compaction requested=true 2023-04-25 06:54:54,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:54,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:54,305 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,306 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,306 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:54,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@565996b5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:54,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:54,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:54,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:54,315 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:54,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:54,319 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:54,319 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:54,319 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:54,319 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:54,319 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:54,325 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,326 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,326 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,326 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:54,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:54,332 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:54,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,332 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,333 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,332 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:54,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:54,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:54:54,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:54,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:54,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:54,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:54,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:54,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:54,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/44cc01e8f23c4af2bc9cee2017e5a10c 2023-04-25 06:54:54,397 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/d7610d1d71664cbb8db9829895733945 2023-04-25 06:54:54,433 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:54,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:54,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:54,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:54,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:54,436 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:54,436 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,436 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:54,436 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:54,437 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,437 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,438 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,438 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,438 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,439 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:54,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:54,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:54,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:54,440 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,440 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:54,440 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:54,440 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:54,441 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:54,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57d8eb0a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:54,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:54,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,445 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:54,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405694410, max timestamp=1682405694443Num uniques 859; 2023-04-25 06:54:54,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405694410, max timestamp=1682405694443Num uniques 859; 2023-04-25 06:54:54,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:54,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:54,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405694410, max timestamp=1682405694443Num uniques 859; 2023-04-25 06:54:54,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:54,446 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,446 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:54,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:54,450 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:54,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:54,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:54,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:54,450 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:54,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,453 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,454 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,456 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:54,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:54,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,457 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,457 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,457 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:54,457 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:54,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682405694465, max timestamp=1682405694500Num uniques 1018; 2023-04-25 06:54:54,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682405694465, max timestamp=1682405694500Num uniques 1018; 2023-04-25 06:54:54,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-25 06:54:54,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682405694492, max timestamp=1682405694500Num uniques 1018; 2023-04-25 06:54:54,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-25 06:54:54,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 06:54:54,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405754515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405754516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405754517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405754519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405754520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,557 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:54,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:54,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:54,560 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:54,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:54,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:54,561 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,561 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:54,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:54,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:54,564 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,564 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:54,564 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:54,564 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:54,567 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:54,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cb769e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:54,569 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:54,569 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,570 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:54,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:54,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:54,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:54,575 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:54,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:54,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:54,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:54,575 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:54,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:54,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:54,584 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:54,584 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:54,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405754621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405754621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405754621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405754623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405754624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,685 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:54,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:54,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:54,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:54,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:54,688 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:54,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:54,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:54,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,693 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,693 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:54,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:54,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:54,695 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,695 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:54,695 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:54,696 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,697 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:54,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cbca71[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:54,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:54,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,700 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:54,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:54,703 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:54,706 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:54,706 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:54,706 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:54,706 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:54,706 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:54,707 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,707 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,707 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,709 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,714 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:54,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,714 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,714 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:54,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:54,714 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:54,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/26350b5425cc451481c412ea05caa061 2023-04-25 06:54:54,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/44cc01e8f23c4af2bc9cee2017e5a10c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/44cc01e8f23c4af2bc9cee2017e5a10c 2023-04-25 06:54:54,815 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:54,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/44cc01e8f23c4af2bc9cee2017e5a10c, entries=200, sequenceid=536, filesize=14.3 K 2023-04-25 06:54:54,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:54,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/d7610d1d71664cbb8db9829895733945 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/d7610d1d71664cbb8db9829895733945 2023-04-25 06:54:54,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:54,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:54,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:54,818 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:54,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:54,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:54,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/d7610d1d71664cbb8db9829895733945, entries=150, sequenceid=536, filesize=11.9 K 2023-04-25 06:54:54,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/26350b5425cc451481c412ea05caa061 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26350b5425cc451481c412ea05caa061 2023-04-25 06:54:54,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:54,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:54,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:54,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:54,823 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,823 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:54,823 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:54,823 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:54,824 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,826 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26350b5425cc451481c412ea05caa061, entries=150, sequenceid=536, filesize=11.9 K 2023-04-25 06:54:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.65 KB/141980 for 11919fcb635bcaf9ea0c7782550d9d17 in 446ms, sequenceid=536, compaction requested=true 2023-04-25 06:54:54,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:54,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:54,828 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:54:54,828 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:54:54,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:54,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:54,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:54,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=145.36 KB heapSize=404.84 KB 2023-04-25 06:54:54,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:54,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:54,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:54,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:54,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:54,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:54,830 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73488 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:54:54,830 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/A is initiating minor compaction (all files) 2023-04-25 06:54:54,830 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71048 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:54:54,830 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/A in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,830 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/B is initiating minor compaction (all files) 2023-04-25 06:54:54,830 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/54702c21b2314da3b8e64e56d3d1bcd8, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/38825da17c0840698f6c5ca1db354774, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/32605f0d975947e2a9d956145f1d4b11, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/10be9d8f7b1d4f6fba0462a15e3a2fcf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/564f13e322d34aad8e32eacf4461d121, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/44cc01e8f23c4af2bc9cee2017e5a10c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=71.8 K 2023-04-25 06:54:54,830 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/B in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,830 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bb77516f48c41768659a06ba04921c6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ef037edda9094a06abfe8cdebc6fc818, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/217def7fef7a4618b8f0e8e546d194cb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/a66feae88d004d9f91fa1ec990c97a9a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c84b267a78c459d9be7ef6c23ee90a4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/d7610d1d71664cbb8db9829895733945] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=69.4 K 2023-04-25 06:54:54,830 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 54702c21b2314da3b8e64e56d3d1bcd8, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682405689880 2023-04-25 06:54:54,831 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4bb77516f48c41768659a06ba04921c6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682405689880 2023-04-25 06:54:54,831 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 38825da17c0840698f6c5ca1db354774, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682405689955 2023-04-25 06:54:54,831 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ef037edda9094a06abfe8cdebc6fc818, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682405689955 2023-04-25 06:54:54,831 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 32605f0d975947e2a9d956145f1d4b11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682405691078 2023-04-25 06:54:54,831 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 10be9d8f7b1d4f6fba0462a15e3a2fcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682405692222 2023-04-25 06:54:54,831 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 217def7fef7a4618b8f0e8e546d194cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682405691078 2023-04-25 06:54:54,832 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 564f13e322d34aad8e32eacf4461d121, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682405693343 2023-04-25 06:54:54,832 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a66feae88d004d9f91fa1ec990c97a9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682405692222 2023-04-25 06:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,832 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 44cc01e8f23c4af2bc9cee2017e5a10c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682405694369 2023-04-25 06:54:54,832 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3c84b267a78c459d9be7ef6c23ee90a4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682405693343 2023-04-25 06:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@283d1ea3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:54,833 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting d7610d1d71664cbb8db9829895733945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682405694373 2023-04-25 06:54:54,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:54,833 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:54,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,835 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:54,836 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:54,839 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:54,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:54,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:54,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:54,839 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,843 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:54,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:54,846 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#A#compaction#240 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:54,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,847 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:54,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:54,847 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,850 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#B#compaction#242 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:54,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405754852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405754853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405754853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405754854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405754854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,873 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/18eb5f63785942c9838617d96e243230 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/18eb5f63785942c9838617d96e243230 2023-04-25 06:54:54,877 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/B of 11919fcb635bcaf9ea0c7782550d9d17 into 18eb5f63785942c9838617d96e243230(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:54,877 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:54,878 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/B, priority=10, startTime=1682405694828; duration=0sec 2023-04-25 06:54:54,878 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:54,878 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:54:54,879 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71048 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:54:54,879 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/C is initiating minor compaction (all files) 2023-04-25 06:54:54,879 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/C in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,879 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/ae1e6c3fb3e24c4d8ebc3354f49f45d3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/648084c7372949758c8159a83dd54d12, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/b593fe7d6e1149d2813189817b576a8d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/44cf1afc093e4cb29ea5f130d827a54f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/1dc4236735ba4c048545a7e75573abf5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26350b5425cc451481c412ea05caa061] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=69.4 K 2023-04-25 06:54:54,880 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ae1e6c3fb3e24c4d8ebc3354f49f45d3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682405689880 2023-04-25 06:54:54,880 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 648084c7372949758c8159a83dd54d12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682405689955 2023-04-25 06:54:54,880 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b593fe7d6e1149d2813189817b576a8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682405691078 2023-04-25 06:54:54,881 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 44cf1afc093e4cb29ea5f130d827a54f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682405692222 2023-04-25 06:54:54,881 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1dc4236735ba4c048545a7e75573abf5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682405693343 2023-04-25 06:54:54,881 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 26350b5425cc451481c412ea05caa061, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682405694373 2023-04-25 06:54:54,889 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#C#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:54,947 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:54,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:54,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:54,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:54,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:54,950 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:54,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:54,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:54,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:54,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:54,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:54,954 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,954 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:54,954 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:54,954 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:54,956 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:54,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ad2015[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:54,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:54,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,960 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:54,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:54,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:54,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:54,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405754959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405754960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405754960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:54,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:54,962 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405754960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405754961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:54,966 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:54,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:54,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:54,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:54,966 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:54,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:54,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:54,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:54,970 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,972 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:54,972 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:54,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:54,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:54,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:54,973 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:54,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:54,973 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:54,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:54,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:54,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,073 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,077 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,079 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,079 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,082 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,082 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,082 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,082 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,083 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51bbde18[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,086 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,088 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,091 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,091 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,091 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,091 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,091 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,101 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:55,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,102 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,102 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405755162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405755162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405755163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405755164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405755165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,202 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,206 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,206 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,206 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,206 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,206 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,206 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,214 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,218 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4f0e1c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,221 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,221 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,222 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,224 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,226 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,227 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,228 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,228 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,228 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,234 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,234 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,266 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/911d5464a197434d8c807ba677a8894f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/911d5464a197434d8c807ba677a8894f 2023-04-25 06:54:55,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/2a0b2d989987401f8d530037870e7624 2023-04-25 06:54:55,270 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/A of 11919fcb635bcaf9ea0c7782550d9d17 into 911d5464a197434d8c807ba677a8894f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:55,270 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:55,271 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/A, priority=10, startTime=1682405694828; duration=0sec 2023-04-25 06:54:55,271 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:55,297 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/368316753a3644069f1d22a0def623e7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/368316753a3644069f1d22a0def623e7 2023-04-25 06:54:55,303 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/C of 11919fcb635bcaf9ea0c7782550d9d17 into 368316753a3644069f1d22a0def623e7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:55,303 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:55,303 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/C, priority=10, startTime=1682405694828; duration=0sec 2023-04-25 06:54:55,304 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:55,335 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,339 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,340 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,342 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,343 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,343 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,343 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d47c710[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,348 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,351 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,354 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,354 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,356 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,356 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,356 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,357 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,358 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,361 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,361 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,361 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:55,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,461 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,465 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,466 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405755466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,469 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,469 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405755468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405755469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405755469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405755469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,477 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cbbfd25[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,479 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,479 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,480 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,480 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,484 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,484 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,486 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,486 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,486 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,487 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,491 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:55,492 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,492 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,592 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,595 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,595 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,595 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,595 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,596 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,596 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,597 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,597 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,599 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,599 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,599 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,599 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,604 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237fbdb4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,605 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,606 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,608 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,608 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,608 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,608 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,608 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,608 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,610 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,612 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,612 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,615 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,617 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,618 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:55,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,618 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,618 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9572ac0db72149ba8fa06255a3d86bd0 2023-04-25 06:54:55,719 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,722 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,722 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,723 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,723 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,724 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,725 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,725 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,726 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,728 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66056829[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,729 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,730 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,731 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,731 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,736 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,737 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,737 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,737 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,737 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,738 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,738 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,738 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,739 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,740 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,742 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,742 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,742 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:55,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,742 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,742 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,743 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,743 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,743 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,743 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,743 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,743 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,843 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,846 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,848 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,848 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,848 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,850 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,853 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b95cbe[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,854 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,854 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,856 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,856 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,856 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,857 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,857 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,860 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,860 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,860 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,860 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,860 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,862 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,864 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,868 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,868 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,968 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:55,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:55,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:55,971 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:55,971 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,971 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:55,971 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:55,971 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,972 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405755971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,973 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,973 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405755972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:55,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:55,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:55,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:55,976 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:55,976 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:55,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405755974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405755974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405755975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5698cff9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:55,979 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:55,979 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,980 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,980 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,980 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:55,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:55,985 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:55,988 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:55,988 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:55,988 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:55,988 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:55,988 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:55,989 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,990 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:55,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,990 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:55,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:55,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,992 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:55,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:55,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:55,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:55,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:55,995 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:55,995 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:55,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:56,095 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:56,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:56,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/6b9c0cccc5cf46ca8a5742180e98341c 2023-04-25 06:54:56,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:56,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:56,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:56,101 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:56,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:56,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:56,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:56,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/2a0b2d989987401f8d530037870e7624 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/2a0b2d989987401f8d530037870e7624 2023-04-25 06:54:56,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:56,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:56,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:56,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:56,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:56,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:56,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:56,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:56,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:56,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:56,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:56,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:56,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:56,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:56,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:56,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/2a0b2d989987401f8d530037870e7624, entries=250, sequenceid=590, filesize=16.7 K 2023-04-25 06:54:56,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9572ac0db72149ba8fa06255a3d86bd0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9572ac0db72149ba8fa06255a3d86bd0 2023-04-25 06:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,108 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:56,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9572ac0db72149ba8fa06255a3d86bd0, entries=150, sequenceid=590, filesize=11.9 K 2023-04-25 06:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d45a83[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:56,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:56,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/6b9c0cccc5cf46ca8a5742180e98341c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6b9c0cccc5cf46ca8a5742180e98341c 2023-04-25 06:54:56,111 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:56,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:56,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:56,112 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,112 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:56,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:56,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6b9c0cccc5cf46ca8a5742180e98341c, entries=150, sequenceid=590, filesize=11.9 K 2023-04-25 06:54:56,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~424.72 KB/434912, currentSize=40.25 KB/41220 for 11919fcb635bcaf9ea0c7782550d9d17 in 1287ms, sequenceid=590, compaction requested=false 2023-04-25 06:54:56,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:56,125 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:56,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:56,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:56,125 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:56,125 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:56,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:56,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:56,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:56,128 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:56,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:56,133 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:56,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:56,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:56,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:56,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:56,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:56,134 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:56,134 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:56,234 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:56,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:56,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:56,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:56,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:56,238 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:56,238 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:56,238 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:56,238 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:56,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,240 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,240 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:56,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:56,240 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:56,240 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:56,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:56,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:56,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:56,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:56,242 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 06:54:56,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:56,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:56,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:56,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:56,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:56,243 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:56,650 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/40f36874c128474eb2232ed5dc09fc98 2023-04-25 06:54:56,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:56,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:56,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405696987, max timestamp=1682405696993Num uniques 859; 2023-04-25 06:54:56,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:56,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405696987, max timestamp=1682405696993Num uniques 859; 2023-04-25 06:54:56,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:56,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405696987, max timestamp=1682405696993Num uniques 859; 2023-04-25 06:54:56,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:54:57,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405757007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682405697005, max timestamp=1682405697011Num uniques 1018; 2023-04-25 06:54:57,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 06:54:57,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682405697005, max timestamp=1682405697011Num uniques 1018; 2023-04-25 06:54:57,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682405697005, max timestamp=1682405697011Num uniques 1018; 2023-04-25 06:54:57,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 06:54:57,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 06:54:57,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682405697028, max timestamp=1682405697034Num uniques 1027; 2023-04-25 06:54:57,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 06:54:57,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682405697028, max timestamp=1682405697034Num uniques 1027; 2023-04-25 06:54:57,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682405697028, max timestamp=1682405697034Num uniques 1027; 2023-04-25 06:54:57,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 06:54:57,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 06:54:57,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405757052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405757053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405757055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405757056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,061 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/edf1e3763e3046648d31cf1c1588b672 2023-04-25 06:54:57,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405757114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405757156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405757157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405757159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405757159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405757317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405757360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405757360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405757362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405757362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,474 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/65d83797496e4d5d80ae40c0af0b244b 2023-04-25 06:54:57,479 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/40f36874c128474eb2232ed5dc09fc98 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/40f36874c128474eb2232ed5dc09fc98 2023-04-25 06:54:57,483 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/40f36874c128474eb2232ed5dc09fc98, entries=150, sequenceid=602, filesize=11.9 K 2023-04-25 06:54:57,484 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/edf1e3763e3046648d31cf1c1588b672 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/edf1e3763e3046648d31cf1c1588b672 2023-04-25 06:54:57,489 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/edf1e3763e3046648d31cf1c1588b672, entries=150, sequenceid=602, filesize=11.9 K 2023-04-25 06:54:57,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/65d83797496e4d5d80ae40c0af0b244b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/65d83797496e4d5d80ae40c0af0b244b 2023-04-25 06:54:57,493 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/65d83797496e4d5d80ae40c0af0b244b, entries=150, sequenceid=602, filesize=11.9 K 2023-04-25 06:54:57,494 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 11919fcb635bcaf9ea0c7782550d9d17 in 1251ms, sequenceid=602, compaction requested=true 2023-04-25 06:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:57,495 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45dd3f90[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:57,497 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:57,497 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,498 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:57,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,499 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:57,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:57,504 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:57,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:57,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:57,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:57,504 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:57,505 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,505 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:57,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,505 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:57,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:57,506 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:57,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:57,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:57,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:57,510 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,510 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:57,510 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:57,510 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:57,610 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:57,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:57,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:57,614 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:57,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:57,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:57,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:57,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:57,615 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,615 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:57,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:57,616 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,616 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:57,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:57,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:57,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:57,617 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:57,617 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:57,617 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=140.89 KB heapSize=398.53 KB 2023-04-25 06:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:57,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:57,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:57,626 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/daaa9a7ebfe7495ea27086142c11b45c 2023-04-25 06:54:57,638 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9229927cfb6d477d98d81065cd71b94e 2023-04-25 06:54:57,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405757639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405757662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405757663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405757665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405757666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405757745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:57,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:57,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405757947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,058 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/8c2a256747d14534b1704dcc20f1f41c 2023-04-25 06:54:58,063 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/daaa9a7ebfe7495ea27086142c11b45c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/daaa9a7ebfe7495ea27086142c11b45c 2023-04-25 06:54:58,066 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/daaa9a7ebfe7495ea27086142c11b45c, entries=150, sequenceid=673, filesize=11.9 K 2023-04-25 06:54:58,067 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9229927cfb6d477d98d81065cd71b94e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9229927cfb6d477d98d81065cd71b94e 2023-04-25 06:54:58,071 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9229927cfb6d477d98d81065cd71b94e, entries=150, sequenceid=673, filesize=11.9 K 2023-04-25 06:54:58,072 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/8c2a256747d14534b1704dcc20f1f41c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8c2a256747d14534b1704dcc20f1f41c 2023-04-25 06:54:58,075 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8c2a256747d14534b1704dcc20f1f41c, entries=150, sequenceid=673, filesize=11.9 K 2023-04-25 06:54:58,076 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~398.48 KB/408048, currentSize=40.25 KB/41220 for 11919fcb635bcaf9ea0c7782550d9d17 in 459ms, sequenceid=673, compaction requested=true 2023-04-25 06:54:58,076 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:58,076 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,079 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aeec739[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,083 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,086 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,086 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,093 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:58,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,097 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:58,098 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:58,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:58,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:54:58,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:58,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:58,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:58,171 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:58,171 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:58,171 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:54:58,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405698185, max timestamp=1682405698190Num uniques 859; 2023-04-25 06:54:58,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:58,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405698185, max timestamp=1682405698190Num uniques 859; 2023-04-25 06:54:58,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405698185, max timestamp=1682405698190Num uniques 859; 2023-04-25 06:54:58,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:58,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:54:58,198 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:58,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:58,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:58,202 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:58,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:58,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:58,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,204 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:58,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:58,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:58,216 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,217 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:58,217 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:58,217 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405758212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405758213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a19229a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405758217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405758218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,228 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,231 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,232 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,233 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,233 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,233 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,235 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,237 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:58,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:58,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,238 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,238 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,238 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,238 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,238 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:58,238 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:58,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405758251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405758319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405758319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405758323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405758323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,338 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:58,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:58,342 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:58,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:58,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:58,343 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,343 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:58,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:58,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:58,347 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,347 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:58,347 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:58,347 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,349 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b36718[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,353 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,354 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,356 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,356 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,356 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,356 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,356 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,359 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,359 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,359 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,359 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,361 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:58,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:58,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:58,363 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,363 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:58,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,363 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:58,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,364 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,364 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,463 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:58,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:58,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:58,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:58,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:58,467 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:58,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:58,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:58,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,469 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,469 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,469 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,469 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:58,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:58,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,471 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:58,471 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:58,471 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c3eeb21[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,474 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,477 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,477 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,478 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,481 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,481 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,483 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,494 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:58,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:58,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:58,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,495 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:58,495 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,495 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:58,495 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,495 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405758523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405758523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405758526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405758527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/686d5602c801497c9e97bfe050e8b13e 2023-04-25 06:54:58,595 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:58,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:58,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:58,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:58,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:58,598 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:58,598 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,598 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:58,598 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:58,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,600 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,600 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,600 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,600 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:58,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:58,601 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:58,602 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,602 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:58,602 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:58,602 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,604 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60b53490[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,608 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,608 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,608 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,611 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,611 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,611 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,613 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,613 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,615 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,615 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,615 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,615 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,619 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,619 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,619 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,619 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,620 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,620 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,620 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,620 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,620 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,620 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:58,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:58,620 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:58,620 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:58,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:58,720 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:58,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:58,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:58,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:58,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:58,724 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:58,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:58,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:58,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,726 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:58,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:58,728 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,728 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:58,728 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:58,728 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,731 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,731 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@134bc7ee[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,734 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,734 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,734 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,734 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,734 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,734 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,735 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,738 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,738 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,739 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,740 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,740 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,740 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,741 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,745 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,745 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:58,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:58,745 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,745 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,745 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,745 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:58,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:58,745 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:58,745 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,746 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,746 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,746 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405758754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405758826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405758828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405758830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405758831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,846 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:58,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:58,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:58,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:58,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:58,849 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:58,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:58,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:58,850 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,850 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,851 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,851 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:58,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:58,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:58,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:58,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18cbf938[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,858 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,858 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,858 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,860 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,863 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,863 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,864 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,864 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,864 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:58,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:58,874 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,874 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:58,874 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:58,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,974 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:58,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:58,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:58,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:58,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:58,978 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:58,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:58,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:58,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,982 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:58,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:58,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:58,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:58,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:58,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:58,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:58,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:58,983 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:58,984 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,984 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:58,984 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:58,984 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:58,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,986 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:58,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53676ee3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:58,988 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:58,988 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,989 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:58,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:58,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e85b8fadb7fe47e1acf4e55478c5d39d 2023-04-25 06:54:58,991 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:58,995 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:58,995 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:58,995 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:58,995 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:58,995 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:58,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:58,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:58,998 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,002 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,002 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,002 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,003 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,002 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,003 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:59,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:59,003 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,003 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,003 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,003 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,003 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:59,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,003 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:59,103 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:59,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:59,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:59,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:59,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:59,107 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:59,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:59,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:59,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,108 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,108 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:59,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:59,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:59,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:59,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:59,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:59,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:59,110 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,110 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:59,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1abfc1ab[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:59,115 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:59,115 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,116 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:59,116 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,117 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,118 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:59,120 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:59,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:59,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:59,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:59,120 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:59,122 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,122 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,122 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,122 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,124 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,127 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,128 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:59,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,128 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:59,128 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:59,228 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:59,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:59,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:59,232 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:59,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:59,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:59,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,233 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:59,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:59,235 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,235 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:59,235 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:59,235 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:59,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:59,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:59,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:59,237 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,237 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,237 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@570611a5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:59,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:59,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,240 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,240 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,240 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:59,240 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:59,240 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:59,240 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:59,242 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:59,245 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:59,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:59,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:59,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:59,245 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:59,246 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,246 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,246 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,246 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,248 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,249 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,249 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,249 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,250 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,250 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,250 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:59,250 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:59,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:59,250 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,250 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,250 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:59,250 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:59,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:59,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405759334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:59,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:59,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405759334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405759334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:59,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405759335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,351 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:59,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:59,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:59,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:59,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:59,354 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:59,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:59,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:59,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,356 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,356 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,356 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,356 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:59,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:59,357 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:59,357 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:59,358 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,358 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:59,358 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:59,358 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:59,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10c32044[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:59,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:59,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:59,366 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,366 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:59,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:59,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:59,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:59,370 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:59,370 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:59,370 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:59,370 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:59,370 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:59,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,371 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,374 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:59,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:59,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:54:59,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,378 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,378 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:59,379 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:59,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/6accc96c839a402d995cabc9677028c2 2023-04-25 06:54:59,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/686d5602c801497c9e97bfe050e8b13e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/686d5602c801497c9e97bfe050e8b13e 2023-04-25 06:54:59,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/686d5602c801497c9e97bfe050e8b13e, entries=150, sequenceid=683, filesize=11.9 K 2023-04-25 06:54:59,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e85b8fadb7fe47e1acf4e55478c5d39d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e85b8fadb7fe47e1acf4e55478c5d39d 2023-04-25 06:54:59,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e85b8fadb7fe47e1acf4e55478c5d39d, entries=100, sequenceid=683, filesize=9.5 K 2023-04-25 06:54:59,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/6accc96c839a402d995cabc9677028c2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6accc96c839a402d995cabc9677028c2 2023-04-25 06:54:59,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6accc96c839a402d995cabc9677028c2, entries=100, sequenceid=683, filesize=9.5 K 2023-04-25 06:54:59,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 11919fcb635bcaf9ea0c7782550d9d17 in 1250ms, sequenceid=683, compaction requested=true 2023-04-25 06:54:59,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:59,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:59,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:59,420 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:59,420 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:59,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:59,422 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:59,422 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59066 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:59,422 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/A is initiating minor compaction (all files) 2023-04-25 06:54:59,422 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/B is initiating minor compaction (all files) 2023-04-25 06:54:59,422 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/A in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,422 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/B in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,422 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/911d5464a197434d8c807ba677a8894f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/2a0b2d989987401f8d530037870e7624, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/40f36874c128474eb2232ed5dc09fc98, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/daaa9a7ebfe7495ea27086142c11b45c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/686d5602c801497c9e97bfe050e8b13e] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=64.8 K 2023-04-25 06:54:59,422 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/18eb5f63785942c9838617d96e243230, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9572ac0db72149ba8fa06255a3d86bd0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/edf1e3763e3046648d31cf1c1588b672, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9229927cfb6d477d98d81065cd71b94e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e85b8fadb7fe47e1acf4e55478c5d39d] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=57.7 K 2023-04-25 06:54:59,422 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 911d5464a197434d8c807ba677a8894f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682405694373 2023-04-25 06:54:59,422 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 18eb5f63785942c9838617d96e243230, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682405694373 2023-04-25 06:54:59,423 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 2a0b2d989987401f8d530037870e7624, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682405694515 2023-04-25 06:54:59,423 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9572ac0db72149ba8fa06255a3d86bd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682405694518 2023-04-25 06:54:59,423 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 40f36874c128474eb2232ed5dc09fc98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682405694852 2023-04-25 06:54:59,423 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting edf1e3763e3046648d31cf1c1588b672, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682405694852 2023-04-25 06:54:59,423 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting daaa9a7ebfe7495ea27086142c11b45c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1682405697046 2023-04-25 06:54:59,424 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9229927cfb6d477d98d81065cd71b94e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1682405697046 2023-04-25 06:54:59,424 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 686d5602c801497c9e97bfe050e8b13e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682405697634 2023-04-25 06:54:59,424 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e85b8fadb7fe47e1acf4e55478c5d39d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682405697637 2023-04-25 06:54:59,435 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#B#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:59,440 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#A#compaction#256 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:59,452 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/c74ce1496ef54efd9c6d4c0a2eb63a08 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c74ce1496ef54efd9c6d4c0a2eb63a08 2023-04-25 06:54:59,458 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/A of 11919fcb635bcaf9ea0c7782550d9d17 into c74ce1496ef54efd9c6d4c0a2eb63a08(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:59,458 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:59,458 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/A, priority=11, startTime=1682405699420; duration=0sec 2023-04-25 06:54:59,458 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:54:59,458 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:54:59,460 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59066 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:54:59,460 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/C is initiating minor compaction (all files) 2023-04-25 06:54:59,460 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/C in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,460 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/368316753a3644069f1d22a0def623e7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6b9c0cccc5cf46ca8a5742180e98341c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/65d83797496e4d5d80ae40c0af0b244b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8c2a256747d14534b1704dcc20f1f41c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6accc96c839a402d995cabc9677028c2] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=57.7 K 2023-04-25 06:54:59,460 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 368316753a3644069f1d22a0def623e7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682405694373 2023-04-25 06:54:59,460 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6b9c0cccc5cf46ca8a5742180e98341c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682405694518 2023-04-25 06:54:59,461 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 65d83797496e4d5d80ae40c0af0b244b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682405694852 2023-04-25 06:54:59,461 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 8c2a256747d14534b1704dcc20f1f41c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1682405697046 2023-04-25 06:54:59,461 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6accc96c839a402d995cabc9677028c2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682405697637 2023-04-25 06:54:59,469 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#C#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:54:59,477 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/382cd7a6cc68419ba5b9e95ad7ea486c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/382cd7a6cc68419ba5b9e95ad7ea486c 2023-04-25 06:54:59,479 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:54:59,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:54:59,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:54:59,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:54:59,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:54:59,482 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:54:59,482 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,482 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:54:59,482 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:54:59,482 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/C of 11919fcb635bcaf9ea0c7782550d9d17 into 382cd7a6cc68419ba5b9e95ad7ea486c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:59,482 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:59,483 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/C, priority=11, startTime=1682405699420; duration=0sec 2023-04-25 06:54:59,483 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:59,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:54:59,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:54:59,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:54:59,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:54:59,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:54:59,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:54:59,489 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:54:59,489 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:54:59,498 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/c6f98bd6dc084fc287a857f37f4bb261 2023-04-25 06:54:59,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:54:59,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:54:59,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:59,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405759795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,846 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9be0d63ec1164fef943b62fe4e4d235e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9be0d63ec1164fef943b62fe4e4d235e 2023-04-25 06:54:59,853 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/B of 11919fcb635bcaf9ea0c7782550d9d17 into 9be0d63ec1164fef943b62fe4e4d235e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:54:59,853 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:59,854 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/B, priority=11, startTime=1682405699420; duration=0sec 2023-04-25 06:54:59,854 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:54:59,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:54:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405759898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,909 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/ed37beb8d53e4894a62473207f1252ee 2023-04-25 06:54:59,936 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/4266feff2a214a88b8840430fda4106a 2023-04-25 06:54:59,943 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/c6f98bd6dc084fc287a857f37f4bb261 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c6f98bd6dc084fc287a857f37f4bb261 2023-04-25 06:54:59,949 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c6f98bd6dc084fc287a857f37f4bb261, entries=150, sequenceid=721, filesize=11.9 K 2023-04-25 06:54:59,950 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/ed37beb8d53e4894a62473207f1252ee as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ed37beb8d53e4894a62473207f1252ee 2023-04-25 06:54:59,955 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ed37beb8d53e4894a62473207f1252ee, entries=150, sequenceid=721, filesize=11.9 K 2023-04-25 06:54:59,956 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/4266feff2a214a88b8840430fda4106a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4266feff2a214a88b8840430fda4106a 2023-04-25 06:54:59,961 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4266feff2a214a88b8840430fda4106a, entries=150, sequenceid=721, filesize=11.9 K 2023-04-25 06:54:59,962 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 11919fcb635bcaf9ea0c7782550d9d17 in 473ms, sequenceid=721, compaction requested=false 2023-04-25 06:54:59,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:54:59,962 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:54:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:54:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:54:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:54:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:54:59,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:54:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78bb54c9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:54:59,967 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:54:59,967 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,968 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:54:59,968 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:54:59,968 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:54:59,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:54:59,970 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:54:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:54:59,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:54:59,974 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:54:59,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:54:59,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:54:59,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:54:59,974 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:54:59,977 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:54:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:54:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:54:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,979 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:54:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:54:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:54:59,982 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,982 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,982 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:54:59,982 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:54:59,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:54:59,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:54:59,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:54:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:54:59,983 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:54:59,982 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:54:59,983 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,983 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:54:59,983 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:54:59,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:54:59,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:54:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:54:59,983 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:54:59,983 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:54:59,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:00,083 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:00,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:00,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:00,087 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:00,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:00,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:00,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:00,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:00,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:00,091 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:55:00,092 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:55:00,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:55:00,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:00,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:55:00,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:00,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:55:00,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:00,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:00,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:55:00,147 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/784ea24f9b5f4c9b9d6310411ac8b715 2023-04-25 06:55:00,170 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e55c3d91e9e14553903059b17c83ff7b 2023-04-25 06:55:00,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405700155, max timestamp=1682405700183Num uniques 859; 2023-04-25 06:55:00,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405700155, max timestamp=1682405700183Num uniques 859; 2023-04-25 06:55:00,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405700155, max timestamp=1682405700183Num uniques 859; 2023-04-25 06:55:00,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:55:00,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:55:00,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:55:00,203 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/5b308ffd55b64041ad1fc611e227fd25 2023-04-25 06:55:00,209 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/784ea24f9b5f4c9b9d6310411ac8b715 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/784ea24f9b5f4c9b9d6310411ac8b715 2023-04-25 06:55:00,214 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/784ea24f9b5f4c9b9d6310411ac8b715, entries=150, sequenceid=732, filesize=11.9 K 2023-04-25 06:55:00,215 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/e55c3d91e9e14553903059b17c83ff7b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e55c3d91e9e14553903059b17c83ff7b 2023-04-25 06:55:00,220 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e55c3d91e9e14553903059b17c83ff7b, entries=150, sequenceid=732, filesize=11.9 K 2023-04-25 06:55:00,221 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/5b308ffd55b64041ad1fc611e227fd25 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5b308ffd55b64041ad1fc611e227fd25 2023-04-25 06:55:00,226 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5b308ffd55b64041ad1fc611e227fd25, entries=150, sequenceid=732, filesize=11.9 K 2023-04-25 06:55:00,226 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=80.51 KB/82440 for 11919fcb635bcaf9ea0c7782550d9d17 in 134ms, sequenceid=732, compaction requested=true 2023-04-25 06:55:00,227 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:00,227 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:00,230 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,231 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,231 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=87.22 KB heapSize=245.53 KB 2023-04-25 06:55:00,232 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:55:00,232 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,232 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:55:00,232 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:00,232 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:55:00,232 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:00,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c6d60ec[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:00,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:00,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:00,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:00,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:00,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:00,241 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:00,241 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:00,241 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:00,241 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:00,241 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:00,242 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,242 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,242 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,243 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,244 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,248 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,249 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:00,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,249 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:00,249 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:00,249 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:00,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/1f03d7a2e7bf4e898d8dc909877662db 2023-04-25 06:55:00,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/c795ac9580234d24a66c85f5edcaae49 2023-04-25 06:55:00,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405760310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405760338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405760339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405760342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,349 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:00,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405760346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:00,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:00,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:00,353 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:00,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:00,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:00,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:00,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:00,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:00,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:00,356 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,356 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:55:00,356 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:55:00,356 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,357 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:00,357 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:00,358 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,358 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,359 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,359 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:00,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b81c1fd[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:00,360 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:00,360 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,361 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:00,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:00,361 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,362 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:00,362 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:00,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:00,370 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:00,370 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:00,370 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:00,370 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:00,371 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:00,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,372 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,375 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,378 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,378 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:00,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,379 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:00,379 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:00,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405760416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,479 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:00,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:00,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:00,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:00,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:00,482 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:00,482 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:00,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:00,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,485 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:00,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:00,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:00,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:00,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:00,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:00,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:00,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:00,486 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:00,486 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,486 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:55:00,486 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:55:00,486 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,487 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:00,487 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:00,487 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,487 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:00,487 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:00,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,489 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,489 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:00,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:00,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c91fc9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:00,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:00,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:00,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:00,493 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,493 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:00,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:00,496 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:00,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:00,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:00,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:00,497 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:00,498 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,498 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,498 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,500 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,512 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:00,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:00,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:00,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,513 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,513 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:00,513 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:00,613 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:00,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:00,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:00,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:00,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:00,617 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:00,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:00,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:00,618 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,618 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,619 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,619 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,619 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,619 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:00,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:00,621 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,621 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:55:00,621 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:55:00,621 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:00,623 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:00,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405760620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:00,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f61045[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:00,625 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:00,625 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,626 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:00,627 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,627 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:00,630 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,630 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:00,630 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,630 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:00,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:00,633 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:00,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:00,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:00,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:00,633 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:00,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:00,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:00,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,636 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,637 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,639 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,639 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,639 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:00,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:00,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:00,639 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,639 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:00,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,639 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,640 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:00,640 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:00,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:00,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:00,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:00,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/83762b99baa54b8bb4dbdae10122ea87 2023-04-25 06:55:00,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/1f03d7a2e7bf4e898d8dc909877662db as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/1f03d7a2e7bf4e898d8dc909877662db 2023-04-25 06:55:00,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/1f03d7a2e7bf4e898d8dc909877662db, entries=150, sequenceid=762, filesize=11.9 K 2023-04-25 06:55:00,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/c795ac9580234d24a66c85f5edcaae49 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/c795ac9580234d24a66c85f5edcaae49 2023-04-25 06:55:00,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/c795ac9580234d24a66c85f5edcaae49, entries=150, sequenceid=762, filesize=11.9 K 2023-04-25 06:55:00,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/83762b99baa54b8bb4dbdae10122ea87 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/83762b99baa54b8bb4dbdae10122ea87 2023-04-25 06:55:00,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/83762b99baa54b8bb4dbdae10122ea87, entries=150, sequenceid=762, filesize=11.9 K 2023-04-25 06:55:00,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~245.48 KB/251376, currentSize=93.93 KB/96180 for 11919fcb635bcaf9ea0c7782550d9d17 in 491ms, sequenceid=762, compaction requested=true 2023-04-25 06:55:00,723 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:00,724 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:00,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:00,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:00,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:55:00,724 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:00,725 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:00,725 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/A is initiating minor compaction (all files) 2023-04-25 06:55:00,725 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/A in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,725 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c74ce1496ef54efd9c6d4c0a2eb63a08, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c6f98bd6dc084fc287a857f37f4bb261, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/784ea24f9b5f4c9b9d6310411ac8b715, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/1f03d7a2e7bf4e898d8dc909877662db] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=48.3 K 2023-04-25 06:55:00,725 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:00,725 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/B is initiating minor compaction (all files) 2023-04-25 06:55:00,725 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/B in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,726 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9be0d63ec1164fef943b62fe4e4d235e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ed37beb8d53e4894a62473207f1252ee, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e55c3d91e9e14553903059b17c83ff7b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/c795ac9580234d24a66c85f5edcaae49] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=48.3 K 2023-04-25 06:55:00,726 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c74ce1496ef54efd9c6d4c0a2eb63a08, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682405697048 2023-04-25 06:55:00,726 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9be0d63ec1164fef943b62fe4e4d235e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682405697048 2023-04-25 06:55:00,726 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c6f98bd6dc084fc287a857f37f4bb261, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1682405698212 2023-04-25 06:55:00,726 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ed37beb8d53e4894a62473207f1252ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1682405698212 2023-04-25 06:55:00,729 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 784ea24f9b5f4c9b9d6310411ac8b715, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1682405699766 2023-04-25 06:55:00,729 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e55c3d91e9e14553903059b17c83ff7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1682405699766 2023-04-25 06:55:00,730 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c795ac9580234d24a66c85f5edcaae49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1682405700197 2023-04-25 06:55:00,730 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 1f03d7a2e7bf4e898d8dc909877662db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1682405700197 2023-04-25 06:55:00,742 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:00,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:00,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:00,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:00,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:00,746 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:00,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:00,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:00,747 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,747 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,749 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,749 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#B#compaction#267 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:00,750 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#A#compaction#268 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:00,750 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:00,750 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:00,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:00,750 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:00,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:00,752 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:00,753 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:00,753 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:00,765 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:55:00,766 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:55:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:55:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:55:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:55:00,766 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:00,781 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/f31f2bb4219f4b058718791aa6b24694 2023-04-25 06:55:00,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:00,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:55:00,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:00,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405760966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405761071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,166 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/4bbe1ad7349543c88901983480ac313e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bbe1ad7349543c88901983480ac313e 2023-04-25 06:55:01,171 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/B of 11919fcb635bcaf9ea0c7782550d9d17 into 4bbe1ad7349543c88901983480ac313e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:01,171 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:01,171 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/B, priority=12, startTime=1682405700724; duration=0sec 2023-04-25 06:55:01,171 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:01,171 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:01,186 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:01,186 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/C is initiating minor compaction (all files) 2023-04-25 06:55:01,186 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/C in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:01,186 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/382cd7a6cc68419ba5b9e95ad7ea486c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4266feff2a214a88b8840430fda4106a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5b308ffd55b64041ad1fc611e227fd25, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/83762b99baa54b8bb4dbdae10122ea87] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=48.3 K 2023-04-25 06:55:01,186 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/89b7c324f8b84ddb96734fb82f5e5004 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/89b7c324f8b84ddb96734fb82f5e5004 2023-04-25 06:55:01,187 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 382cd7a6cc68419ba5b9e95ad7ea486c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682405697048 2023-04-25 06:55:01,187 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4266feff2a214a88b8840430fda4106a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1682405698212 2023-04-25 06:55:01,188 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5b308ffd55b64041ad1fc611e227fd25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1682405699766 2023-04-25 06:55:01,189 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 83762b99baa54b8bb4dbdae10122ea87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1682405700197 2023-04-25 06:55:01,193 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/32b5954f115f4b6985608a0cb8d2e559 2023-04-25 06:55:01,204 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#C#compaction#271 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:01,206 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/A of 11919fcb635bcaf9ea0c7782550d9d17 into 89b7c324f8b84ddb96734fb82f5e5004(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:01,206 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:01,206 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/A, priority=12, startTime=1682405700723; duration=0sec 2023-04-25 06:55:01,206 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:01,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:01,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405761273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:01,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405761578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,617 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/8816dc3a4ebf4c429145adb246b014e2 2023-04-25 06:55:01,620 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/4fdb83530ddf469ca7e7fc494300d832 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4fdb83530ddf469ca7e7fc494300d832 2023-04-25 06:55:01,622 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/f31f2bb4219f4b058718791aa6b24694 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/f31f2bb4219f4b058718791aa6b24694 2023-04-25 06:55:01,628 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/C of 11919fcb635bcaf9ea0c7782550d9d17 into 4fdb83530ddf469ca7e7fc494300d832(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:01,628 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:01,628 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/C, priority=12, startTime=1682405700724; duration=0sec 2023-04-25 06:55:01,628 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/f31f2bb4219f4b058718791aa6b24694, entries=150, sequenceid=779, filesize=11.9 K 2023-04-25 06:55:01,628 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:01,629 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/32b5954f115f4b6985608a0cb8d2e559 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/32b5954f115f4b6985608a0cb8d2e559 2023-04-25 06:55:01,633 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/32b5954f115f4b6985608a0cb8d2e559, entries=150, sequenceid=779, filesize=11.9 K 2023-04-25 06:55:01,634 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/8816dc3a4ebf4c429145adb246b014e2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8816dc3a4ebf4c429145adb246b014e2 2023-04-25 06:55:01,637 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8816dc3a4ebf4c429145adb246b014e2, entries=150, sequenceid=779, filesize=11.9 K 2023-04-25 06:55:01,638 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 11919fcb635bcaf9ea0c7782550d9d17 in 872ms, sequenceid=779, compaction requested=false 2023-04-25 06:55:01,638 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:01,638 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:01,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:01,644 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:01,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:01,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@276b70b7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:01,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:01,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:01,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:01,649 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:01,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:01,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:01,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:01,652 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:01,652 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:01,652 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:01,652 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:01,652 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:01,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:01,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:01,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,665 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:01,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:01,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:01,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:01,669 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:01,669 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:01,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:01,670 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:01,769 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:01,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:01,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:01,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:01,773 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:01,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:01,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:01,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:01,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:01,774 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:01,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:01,775 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:01,775 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:01,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:01,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:01,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:01,776 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:01,776 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. started... 2023-04-25 06:55:01,776 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:55:01,777 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:55:01,777 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:01,777 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:55:01,777 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:01,777 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:55:01,777 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:02,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:02,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. as already flushing 2023-04-25 06:55:02,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405762121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,185 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=797 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/441d12417e844fbeb4619e866fa4fad4 2023-04-25 06:55:02,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405762224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51322 deadline: 1682405762352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51280 deadline: 1682405762353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,355 DEBUG [Thread-1560] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:02,356 DEBUG [Thread-1556] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:02,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51278 deadline: 1682405762359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,362 DEBUG [Thread-1558] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:02,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51348 deadline: 1682405762364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,367 DEBUG [Thread-1564] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:02,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405762427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,595 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=797 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9f0752fe3f41488fa79d19f791e10e9f 2023-04-25 06:55:02,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:02,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51342 deadline: 1682405762730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:02,804 DEBUG [Thread-1575] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45de5ad8 to 127.0.0.1:60416 2023-04-25 06:55:02,805 DEBUG [Thread-1575] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:02,805 DEBUG [Thread-1567] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e5b1c14 to 127.0.0.1:60416 2023-04-25 06:55:02,805 DEBUG [Thread-1567] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:02,806 DEBUG [Thread-1569] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f29c1c4 to 127.0.0.1:60416 2023-04-25 06:55:02,806 DEBUG [Thread-1569] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:02,806 DEBUG [Thread-1571] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681ace6c to 127.0.0.1:60416 2023-04-25 06:55:02,807 DEBUG [Thread-1571] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:02,807 DEBUG [Thread-1573] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b4fa5d3 to 127.0.0.1:60416 2023-04-25 06:55:02,807 DEBUG [Thread-1573] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:03,006 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=797 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/63ac5df342f0492c85a2d7fa97b37e4c 2023-04-25 06:55:03,010 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/441d12417e844fbeb4619e866fa4fad4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/441d12417e844fbeb4619e866fa4fad4 2023-04-25 06:55:03,013 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/441d12417e844fbeb4619e866fa4fad4, entries=150, sequenceid=797, filesize=11.9 K 2023-04-25 06:55:03,014 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/9f0752fe3f41488fa79d19f791e10e9f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9f0752fe3f41488fa79d19f791e10e9f 2023-04-25 06:55:03,017 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9f0752fe3f41488fa79d19f791e10e9f, entries=150, sequenceid=797, filesize=11.9 K 2023-04-25 06:55:03,018 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/63ac5df342f0492c85a2d7fa97b37e4c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/63ac5df342f0492c85a2d7fa97b37e4c 2023-04-25 06:55:03,021 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/63ac5df342f0492c85a2d7fa97b37e4c, entries=150, sequenceid=797, filesize=11.9 K 2023-04-25 06:55:03,022 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 11919fcb635bcaf9ea0c7782550d9d17 in 1246ms, sequenceid=797, compaction requested=true 2023-04-25 06:55:03,022 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:03,022 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:03,023 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,023 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:03,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:03,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5210c38e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:03,025 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:03,025 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,027 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:03,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:03,028 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:03,030 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:03,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:03,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:03,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:03,030 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:03,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:03,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:03,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:03,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:03,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,033 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:03,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:03,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:03,034 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:03,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:03,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:03,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:03,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:03,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:03,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:03,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,035 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:03,035 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:03,035 DEBUG [Thread-1566] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:03,035 DEBUG [Thread-1566] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:03,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:03,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:03,135 DEBUG [Thread-1566] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:03,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:03,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:03,237 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:55:03,237 DEBUG [Thread-1562] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3562c7bf to 127.0.0.1:60416 2023-04-25 06:55:03,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:55:03,237 DEBUG [Thread-1562] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:03,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:03,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:55:03,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:03,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:55:03,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:03,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/c8b8d20a6e1742d19d26f0b6a6fc35a4 2023-04-25 06:55:04,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/2f92ae24a5714218a8f0a2cb53593ad8 2023-04-25 06:55:04,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/5a1454e16ea8449ba0b8a93d224bbe52 2023-04-25 06:55:04,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/c8b8d20a6e1742d19d26f0b6a6fc35a4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c8b8d20a6e1742d19d26f0b6a6fc35a4 2023-04-25 06:55:04,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c8b8d20a6e1742d19d26f0b6a6fc35a4, entries=150, sequenceid=815, filesize=11.9 K 2023-04-25 06:55:04,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/2f92ae24a5714218a8f0a2cb53593ad8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2f92ae24a5714218a8f0a2cb53593ad8 2023-04-25 06:55:04,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2f92ae24a5714218a8f0a2cb53593ad8, entries=150, sequenceid=815, filesize=11.9 K 2023-04-25 06:55:04,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/5a1454e16ea8449ba0b8a93d224bbe52 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5a1454e16ea8449ba0b8a93d224bbe52 2023-04-25 06:55:04,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5a1454e16ea8449ba0b8a93d224bbe52, entries=150, sequenceid=815, filesize=11.9 K 2023-04-25 06:55:04,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 11919fcb635bcaf9ea0c7782550d9d17 in 1241ms, sequenceid=815, compaction requested=true 2023-04-25 06:55:04,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:04,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:04,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:04,478 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:04,478 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:04,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:04,479 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:04,479 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/A is initiating minor compaction (all files) 2023-04-25 06:55:04,479 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:04,479 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/A in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:04,479 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/B is initiating minor compaction (all files) 2023-04-25 06:55:04,480 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/89b7c324f8b84ddb96734fb82f5e5004, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/f31f2bb4219f4b058718791aa6b24694, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/441d12417e844fbeb4619e866fa4fad4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c8b8d20a6e1742d19d26f0b6a6fc35a4] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=48.5 K 2023-04-25 06:55:04,480 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/B in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:04,480 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bbe1ad7349543c88901983480ac313e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/32b5954f115f4b6985608a0cb8d2e559, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9f0752fe3f41488fa79d19f791e10e9f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2f92ae24a5714218a8f0a2cb53593ad8] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=48.5 K 2023-04-25 06:55:04,480 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 89b7c324f8b84ddb96734fb82f5e5004, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1682405700197 2023-04-25 06:55:04,480 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4bbe1ad7349543c88901983480ac313e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1682405700197 2023-04-25 06:55:04,480 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting f31f2bb4219f4b058718791aa6b24694, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1682405700299 2023-04-25 06:55:04,480 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 32b5954f115f4b6985608a0cb8d2e559, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1682405700299 2023-04-25 06:55:04,481 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 441d12417e844fbeb4619e866fa4fad4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=797, earliestPutTs=1682405700956 2023-04-25 06:55:04,481 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9f0752fe3f41488fa79d19f791e10e9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=797, earliestPutTs=1682405700956 2023-04-25 06:55:04,481 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c8b8d20a6e1742d19d26f0b6a6fc35a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1682405702114 2023-04-25 06:55:04,481 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2f92ae24a5714218a8f0a2cb53593ad8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1682405702114 2023-04-25 06:55:04,490 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#B#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:04,492 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#A#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:04,500 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/3b6a0e996913423583012ff9069d38c0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b6a0e996913423583012ff9069d38c0 2023-04-25 06:55:04,504 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/A of 11919fcb635bcaf9ea0c7782550d9d17 into 3b6a0e996913423583012ff9069d38c0(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:04,504 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:04,504 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/A, priority=12, startTime=1682405704478; duration=0sec 2023-04-25 06:55:04,504 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:04,504 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:04,505 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:04,505 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 11919fcb635bcaf9ea0c7782550d9d17/C is initiating minor compaction (all files) 2023-04-25 06:55:04,505 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 11919fcb635bcaf9ea0c7782550d9d17/C in TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:04,505 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4fdb83530ddf469ca7e7fc494300d832, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8816dc3a4ebf4c429145adb246b014e2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/63ac5df342f0492c85a2d7fa97b37e4c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5a1454e16ea8449ba0b8a93d224bbe52] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp, totalSize=48.5 K 2023-04-25 06:55:04,505 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 4fdb83530ddf469ca7e7fc494300d832, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1682405700197 2023-04-25 06:55:04,505 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 8816dc3a4ebf4c429145adb246b014e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1682405700299 2023-04-25 06:55:04,506 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 63ac5df342f0492c85a2d7fa97b37e4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=797, earliestPutTs=1682405700956 2023-04-25 06:55:04,506 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5a1454e16ea8449ba0b8a93d224bbe52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1682405702114 2023-04-25 06:55:04,512 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11919fcb635bcaf9ea0c7782550d9d17#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:04,899 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/db49e563d99342c28c24650ea4544419 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/db49e563d99342c28c24650ea4544419 2023-04-25 06:55:04,903 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/B of 11919fcb635bcaf9ea0c7782550d9d17 into db49e563d99342c28c24650ea4544419(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:04,903 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:04,903 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/B, priority=12, startTime=1682405704478; duration=0sec 2023-04-25 06:55:04,903 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:04,920 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/74a96bc19adb4308b2e31ddae6fa90de as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/74a96bc19adb4308b2e31ddae6fa90de 2023-04-25 06:55:04,925 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 11919fcb635bcaf9ea0c7782550d9d17/C of 11919fcb635bcaf9ea0c7782550d9d17 into 74a96bc19adb4308b2e31ddae6fa90de(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:04,925 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:04,925 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17., storeName=11919fcb635bcaf9ea0c7782550d9d17/C, priority=12, startTime=1682405704478; duration=0sec 2023-04-25 06:55:04,925 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:06,382 DEBUG [Thread-1560] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5a81fa to 127.0.0.1:60416 2023-04-25 06:55:06,383 DEBUG [Thread-1560] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:06,389 DEBUG [Thread-1558] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b79c2c to 127.0.0.1:60416 2023-04-25 06:55:06,389 DEBUG [Thread-1558] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:06,395 DEBUG [Thread-1556] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13544701 to 127.0.0.1:60416 2023-04-25 06:55:06,395 DEBUG [Thread-1556] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:06,400 DEBUG [Thread-1564] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6705d115 to 127.0.0.1:60416 2023-04-25 06:55:06,400 DEBUG [Thread-1564] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 177 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 141 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 139 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 5834 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 5573 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 5384 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 5844 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 5556 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 06:55:06,400 INFO [Listener at localhost/41663] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 06:55:06,400 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53921272 to 127.0.0.1:60416 2023-04-25 06:55:06,400 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:06,401 INFO [Listener at localhost/41663] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 06:55:06,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 06:55:06,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:06,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 06:55:06,404 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405706404"}]},"ts":"1682405706404"} 2023-04-25 06:55:06,405 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 06:55:06,406 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 06:55:06,407 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=11919fcb635bcaf9ea0c7782550d9d17, UNASSIGN}] 2023-04-25 06:55:06,408 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=11919fcb635bcaf9ea0c7782550d9d17, UNASSIGN 2023-04-25 06:55:06,409 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=11919fcb635bcaf9ea0c7782550d9d17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:06,409 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405706409"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405706409"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405706409"}]},"ts":"1682405706409"} 2023-04-25 06:55:06,410 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:55:06,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 06:55:06,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 11919fcb635bcaf9ea0c7782550d9d17, disabling compactions & flushes 2023-04-25 06:55:06,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. after waiting 0 ms 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:06,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 11919fcb635bcaf9ea0c7782550d9d17 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=A 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=B 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 11919fcb635bcaf9ea0c7782550d9d17, store=C 2023-04-25 06:55:06,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:06,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 06:55:06,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/ad215947cb5440afadb22a9a5986417f 2023-04-25 06:55:07,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 06:55:07,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/0228d5bbbab0451580166896d60c7a72 2023-04-25 06:55:07,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 06:55:07,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/cbeef5c8e67a46bc98c27ee6077d2618 2023-04-25 06:55:07,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/A/ad215947cb5440afadb22a9a5986417f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ad215947cb5440afadb22a9a5986417f 2023-04-25 06:55:07,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ad215947cb5440afadb22a9a5986417f, entries=100, sequenceid=825, filesize=9.5 K 2023-04-25 06:55:07,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/B/0228d5bbbab0451580166896d60c7a72 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/0228d5bbbab0451580166896d60c7a72 2023-04-25 06:55:07,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/0228d5bbbab0451580166896d60c7a72, entries=100, sequenceid=825, filesize=9.5 K 2023-04-25 06:55:07,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/.tmp/C/cbeef5c8e67a46bc98c27ee6077d2618 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cbeef5c8e67a46bc98c27ee6077d2618 2023-04-25 06:55:07,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cbeef5c8e67a46bc98c27ee6077d2618, entries=100, sequenceid=825, filesize=9.5 K 2023-04-25 06:55:07,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 11919fcb635bcaf9ea0c7782550d9d17 in 1242ms, sequenceid=825, compaction requested=false 2023-04-25 06:55:07,825 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/5598acc8c8b543579e3861912dfde771, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/8d409beef1de41df9a48a026065bd337, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/696bacb104c94672a3de01056818bc08, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c72b299468af4d0eb466fe4143e07293, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/e84b366603264f0d82fc7656650f4809, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0aac9d48011241119b125c1e47047d57, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b5422bf3e1c436183391160f6f57a1a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ca7066f7b9dc436fa931432b8a5acbdf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/655f6efb836845cc9655f1836e664f42, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/227e012bb18540cdad88dcc6c0181879, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/54702c21b2314da3b8e64e56d3d1bcd8, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0591aac787b044a79386300f2ac06ffb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/38825da17c0840698f6c5ca1db354774, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/32605f0d975947e2a9d956145f1d4b11, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/10be9d8f7b1d4f6fba0462a15e3a2fcf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/564f13e322d34aad8e32eacf4461d121, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/44cc01e8f23c4af2bc9cee2017e5a10c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/911d5464a197434d8c807ba677a8894f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/2a0b2d989987401f8d530037870e7624, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/40f36874c128474eb2232ed5dc09fc98, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/daaa9a7ebfe7495ea27086142c11b45c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c74ce1496ef54efd9c6d4c0a2eb63a08, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/686d5602c801497c9e97bfe050e8b13e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c6f98bd6dc084fc287a857f37f4bb261, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/784ea24f9b5f4c9b9d6310411ac8b715, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/89b7c324f8b84ddb96734fb82f5e5004, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/1f03d7a2e7bf4e898d8dc909877662db, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/f31f2bb4219f4b058718791aa6b24694, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/441d12417e844fbeb4619e866fa4fad4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c8b8d20a6e1742d19d26f0b6a6fc35a4] to archive 2023-04-25 06:55:07,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:07,827 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/5598acc8c8b543579e3861912dfde771 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/5598acc8c8b543579e3861912dfde771 2023-04-25 06:55:07,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/8d409beef1de41df9a48a026065bd337 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/8d409beef1de41df9a48a026065bd337 2023-04-25 06:55:07,829 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/696bacb104c94672a3de01056818bc08 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/696bacb104c94672a3de01056818bc08 2023-04-25 06:55:07,830 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c72b299468af4d0eb466fe4143e07293 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c72b299468af4d0eb466fe4143e07293 2023-04-25 06:55:07,831 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/e84b366603264f0d82fc7656650f4809 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/e84b366603264f0d82fc7656650f4809 2023-04-25 06:55:07,832 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0aac9d48011241119b125c1e47047d57 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0aac9d48011241119b125c1e47047d57 2023-04-25 06:55:07,832 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b5422bf3e1c436183391160f6f57a1a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b5422bf3e1c436183391160f6f57a1a 2023-04-25 06:55:07,833 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ca7066f7b9dc436fa931432b8a5acbdf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ca7066f7b9dc436fa931432b8a5acbdf 2023-04-25 06:55:07,834 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/655f6efb836845cc9655f1836e664f42 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/655f6efb836845cc9655f1836e664f42 2023-04-25 06:55:07,835 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/227e012bb18540cdad88dcc6c0181879 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/227e012bb18540cdad88dcc6c0181879 2023-04-25 06:55:07,836 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/54702c21b2314da3b8e64e56d3d1bcd8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/54702c21b2314da3b8e64e56d3d1bcd8 2023-04-25 06:55:07,837 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0591aac787b044a79386300f2ac06ffb to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/0591aac787b044a79386300f2ac06ffb 2023-04-25 06:55:07,838 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/38825da17c0840698f6c5ca1db354774 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/38825da17c0840698f6c5ca1db354774 2023-04-25 06:55:07,839 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/32605f0d975947e2a9d956145f1d4b11 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/32605f0d975947e2a9d956145f1d4b11 2023-04-25 06:55:07,839 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/10be9d8f7b1d4f6fba0462a15e3a2fcf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/10be9d8f7b1d4f6fba0462a15e3a2fcf 2023-04-25 06:55:07,840 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/564f13e322d34aad8e32eacf4461d121 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/564f13e322d34aad8e32eacf4461d121 2023-04-25 06:55:07,841 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/44cc01e8f23c4af2bc9cee2017e5a10c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/44cc01e8f23c4af2bc9cee2017e5a10c 2023-04-25 06:55:07,842 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/911d5464a197434d8c807ba677a8894f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/911d5464a197434d8c807ba677a8894f 2023-04-25 06:55:07,843 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/2a0b2d989987401f8d530037870e7624 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/2a0b2d989987401f8d530037870e7624 2023-04-25 06:55:07,844 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/40f36874c128474eb2232ed5dc09fc98 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/40f36874c128474eb2232ed5dc09fc98 2023-04-25 06:55:07,845 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/daaa9a7ebfe7495ea27086142c11b45c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/daaa9a7ebfe7495ea27086142c11b45c 2023-04-25 06:55:07,846 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c74ce1496ef54efd9c6d4c0a2eb63a08 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c74ce1496ef54efd9c6d4c0a2eb63a08 2023-04-25 06:55:07,847 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/686d5602c801497c9e97bfe050e8b13e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/686d5602c801497c9e97bfe050e8b13e 2023-04-25 06:55:07,848 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c6f98bd6dc084fc287a857f37f4bb261 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c6f98bd6dc084fc287a857f37f4bb261 2023-04-25 06:55:07,848 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/784ea24f9b5f4c9b9d6310411ac8b715 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/784ea24f9b5f4c9b9d6310411ac8b715 2023-04-25 06:55:07,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/89b7c324f8b84ddb96734fb82f5e5004 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/89b7c324f8b84ddb96734fb82f5e5004 2023-04-25 06:55:07,850 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/1f03d7a2e7bf4e898d8dc909877662db to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/1f03d7a2e7bf4e898d8dc909877662db 2023-04-25 06:55:07,851 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/f31f2bb4219f4b058718791aa6b24694 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/f31f2bb4219f4b058718791aa6b24694 2023-04-25 06:55:07,852 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/441d12417e844fbeb4619e866fa4fad4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/441d12417e844fbeb4619e866fa4fad4 2023-04-25 06:55:07,853 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c8b8d20a6e1742d19d26f0b6a6fc35a4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/c8b8d20a6e1742d19d26f0b6a6fc35a4 2023-04-25 06:55:07,856 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2a6a5c8d34c34d2fbd6e0de8af5e363c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9137b6a8468f46ca8f872f77e02af7fe, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c16d57075564c60989b4c5afff20dce, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7223b3c8085e4e8b8a05f2e100b78192, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e52cbc39862f4846a76498bdde7095f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/65bcd588638546b3b45271d60de56602, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7d026c3eccc8445a820314e191e981f6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ff7a6528957c4e7baa9ebcbcdf6d9e13, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e312277148b24662bdd9d9bb2e11b8ab, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/b87a1cab8a8e4495af7c9a33963e5cd6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bb77516f48c41768659a06ba04921c6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/21600154ecba45dcbc48fb4832f13194, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ef037edda9094a06abfe8cdebc6fc818, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/217def7fef7a4618b8f0e8e546d194cb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/a66feae88d004d9f91fa1ec990c97a9a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c84b267a78c459d9be7ef6c23ee90a4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/18eb5f63785942c9838617d96e243230, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/d7610d1d71664cbb8db9829895733945, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9572ac0db72149ba8fa06255a3d86bd0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/edf1e3763e3046648d31cf1c1588b672, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9229927cfb6d477d98d81065cd71b94e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9be0d63ec1164fef943b62fe4e4d235e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e85b8fadb7fe47e1acf4e55478c5d39d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ed37beb8d53e4894a62473207f1252ee, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e55c3d91e9e14553903059b17c83ff7b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bbe1ad7349543c88901983480ac313e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/c795ac9580234d24a66c85f5edcaae49, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/32b5954f115f4b6985608a0cb8d2e559, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9f0752fe3f41488fa79d19f791e10e9f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2f92ae24a5714218a8f0a2cb53593ad8] to archive 2023-04-25 06:55:07,856 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:07,857 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2a6a5c8d34c34d2fbd6e0de8af5e363c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2a6a5c8d34c34d2fbd6e0de8af5e363c 2023-04-25 06:55:07,858 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9137b6a8468f46ca8f872f77e02af7fe to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9137b6a8468f46ca8f872f77e02af7fe 2023-04-25 06:55:07,859 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c16d57075564c60989b4c5afff20dce to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c16d57075564c60989b4c5afff20dce 2023-04-25 06:55:07,860 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7223b3c8085e4e8b8a05f2e100b78192 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7223b3c8085e4e8b8a05f2e100b78192 2023-04-25 06:55:07,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e52cbc39862f4846a76498bdde7095f7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e52cbc39862f4846a76498bdde7095f7 2023-04-25 06:55:07,862 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/65bcd588638546b3b45271d60de56602 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/65bcd588638546b3b45271d60de56602 2023-04-25 06:55:07,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7d026c3eccc8445a820314e191e981f6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/7d026c3eccc8445a820314e191e981f6 2023-04-25 06:55:07,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ff7a6528957c4e7baa9ebcbcdf6d9e13 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ff7a6528957c4e7baa9ebcbcdf6d9e13 2023-04-25 06:55:07,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e312277148b24662bdd9d9bb2e11b8ab to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e312277148b24662bdd9d9bb2e11b8ab 2023-04-25 06:55:07,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/b87a1cab8a8e4495af7c9a33963e5cd6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/b87a1cab8a8e4495af7c9a33963e5cd6 2023-04-25 06:55:07,866 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bb77516f48c41768659a06ba04921c6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bb77516f48c41768659a06ba04921c6 2023-04-25 06:55:07,866 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/21600154ecba45dcbc48fb4832f13194 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/21600154ecba45dcbc48fb4832f13194 2023-04-25 06:55:07,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ef037edda9094a06abfe8cdebc6fc818 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ef037edda9094a06abfe8cdebc6fc818 2023-04-25 06:55:07,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/217def7fef7a4618b8f0e8e546d194cb to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/217def7fef7a4618b8f0e8e546d194cb 2023-04-25 06:55:07,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/a66feae88d004d9f91fa1ec990c97a9a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/a66feae88d004d9f91fa1ec990c97a9a 2023-04-25 06:55:07,870 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c84b267a78c459d9be7ef6c23ee90a4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/3c84b267a78c459d9be7ef6c23ee90a4 2023-04-25 06:55:07,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/18eb5f63785942c9838617d96e243230 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/18eb5f63785942c9838617d96e243230 2023-04-25 06:55:07,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/d7610d1d71664cbb8db9829895733945 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/d7610d1d71664cbb8db9829895733945 2023-04-25 06:55:07,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9572ac0db72149ba8fa06255a3d86bd0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9572ac0db72149ba8fa06255a3d86bd0 2023-04-25 06:55:07,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/edf1e3763e3046648d31cf1c1588b672 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/edf1e3763e3046648d31cf1c1588b672 2023-04-25 06:55:07,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9229927cfb6d477d98d81065cd71b94e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9229927cfb6d477d98d81065cd71b94e 2023-04-25 06:55:07,875 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9be0d63ec1164fef943b62fe4e4d235e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9be0d63ec1164fef943b62fe4e4d235e 2023-04-25 06:55:07,876 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e85b8fadb7fe47e1acf4e55478c5d39d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e85b8fadb7fe47e1acf4e55478c5d39d 2023-04-25 06:55:07,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ed37beb8d53e4894a62473207f1252ee to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/ed37beb8d53e4894a62473207f1252ee 2023-04-25 06:55:07,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e55c3d91e9e14553903059b17c83ff7b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/e55c3d91e9e14553903059b17c83ff7b 2023-04-25 06:55:07,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bbe1ad7349543c88901983480ac313e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/4bbe1ad7349543c88901983480ac313e 2023-04-25 06:55:07,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/c795ac9580234d24a66c85f5edcaae49 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/c795ac9580234d24a66c85f5edcaae49 2023-04-25 06:55:07,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/32b5954f115f4b6985608a0cb8d2e559 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/32b5954f115f4b6985608a0cb8d2e559 2023-04-25 06:55:07,881 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9f0752fe3f41488fa79d19f791e10e9f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/9f0752fe3f41488fa79d19f791e10e9f 2023-04-25 06:55:07,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2f92ae24a5714218a8f0a2cb53593ad8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/2f92ae24a5714218a8f0a2cb53593ad8 2023-04-25 06:55:07,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cebf695a4ce74faca3a29d17f51f4430, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/75c31984e70c4d8abb71f798b30a4f55, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/bc10b4c54e8a4627a5e2321ec36e6b66, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/fdfbe714f1e94789820aac60da73d172, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/20fed97db3094ed8b7912ec49ac73830, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/9f3f9d3d88f24b0589e70c4b2961cc2c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6c38bfcee0ce42a78cf2f7623a40f773, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26f685b5274f4239a05c523d280d9771, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/0c43f8dfee0146178a96d5e3130226b4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/c0c484dfa26d4fd5bdf23c9c98203705, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/ae1e6c3fb3e24c4d8ebc3354f49f45d3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/67b0ef7b9c27446f913376f2f0796f52, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/648084c7372949758c8159a83dd54d12, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/b593fe7d6e1149d2813189817b576a8d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/44cf1afc093e4cb29ea5f130d827a54f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/1dc4236735ba4c048545a7e75573abf5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/368316753a3644069f1d22a0def623e7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26350b5425cc451481c412ea05caa061, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6b9c0cccc5cf46ca8a5742180e98341c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/65d83797496e4d5d80ae40c0af0b244b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8c2a256747d14534b1704dcc20f1f41c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/382cd7a6cc68419ba5b9e95ad7ea486c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6accc96c839a402d995cabc9677028c2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4266feff2a214a88b8840430fda4106a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5b308ffd55b64041ad1fc611e227fd25, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4fdb83530ddf469ca7e7fc494300d832, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/83762b99baa54b8bb4dbdae10122ea87, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8816dc3a4ebf4c429145adb246b014e2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/63ac5df342f0492c85a2d7fa97b37e4c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5a1454e16ea8449ba0b8a93d224bbe52] to archive 2023-04-25 06:55:07,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:07,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cebf695a4ce74faca3a29d17f51f4430 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cebf695a4ce74faca3a29d17f51f4430 2023-04-25 06:55:07,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/75c31984e70c4d8abb71f798b30a4f55 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/75c31984e70c4d8abb71f798b30a4f55 2023-04-25 06:55:07,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/bc10b4c54e8a4627a5e2321ec36e6b66 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/bc10b4c54e8a4627a5e2321ec36e6b66 2023-04-25 06:55:07,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/fdfbe714f1e94789820aac60da73d172 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/fdfbe714f1e94789820aac60da73d172 2023-04-25 06:55:07,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/20fed97db3094ed8b7912ec49ac73830 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/20fed97db3094ed8b7912ec49ac73830 2023-04-25 06:55:07,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/9f3f9d3d88f24b0589e70c4b2961cc2c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/9f3f9d3d88f24b0589e70c4b2961cc2c 2023-04-25 06:55:07,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6c38bfcee0ce42a78cf2f7623a40f773 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6c38bfcee0ce42a78cf2f7623a40f773 2023-04-25 06:55:07,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26f685b5274f4239a05c523d280d9771 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26f685b5274f4239a05c523d280d9771 2023-04-25 06:55:07,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/0c43f8dfee0146178a96d5e3130226b4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/0c43f8dfee0146178a96d5e3130226b4 2023-04-25 06:55:07,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/c0c484dfa26d4fd5bdf23c9c98203705 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/c0c484dfa26d4fd5bdf23c9c98203705 2023-04-25 06:55:07,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/ae1e6c3fb3e24c4d8ebc3354f49f45d3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/ae1e6c3fb3e24c4d8ebc3354f49f45d3 2023-04-25 06:55:07,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/67b0ef7b9c27446f913376f2f0796f52 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/67b0ef7b9c27446f913376f2f0796f52 2023-04-25 06:55:07,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/648084c7372949758c8159a83dd54d12 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/648084c7372949758c8159a83dd54d12 2023-04-25 06:55:07,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/b593fe7d6e1149d2813189817b576a8d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/b593fe7d6e1149d2813189817b576a8d 2023-04-25 06:55:07,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/44cf1afc093e4cb29ea5f130d827a54f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/44cf1afc093e4cb29ea5f130d827a54f 2023-04-25 06:55:07,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/1dc4236735ba4c048545a7e75573abf5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/1dc4236735ba4c048545a7e75573abf5 2023-04-25 06:55:07,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/368316753a3644069f1d22a0def623e7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/368316753a3644069f1d22a0def623e7 2023-04-25 06:55:07,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26350b5425cc451481c412ea05caa061 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/26350b5425cc451481c412ea05caa061 2023-04-25 06:55:07,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6b9c0cccc5cf46ca8a5742180e98341c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6b9c0cccc5cf46ca8a5742180e98341c 2023-04-25 06:55:07,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/65d83797496e4d5d80ae40c0af0b244b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/65d83797496e4d5d80ae40c0af0b244b 2023-04-25 06:55:07,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8c2a256747d14534b1704dcc20f1f41c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8c2a256747d14534b1704dcc20f1f41c 2023-04-25 06:55:07,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/382cd7a6cc68419ba5b9e95ad7ea486c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/382cd7a6cc68419ba5b9e95ad7ea486c 2023-04-25 06:55:07,905 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6accc96c839a402d995cabc9677028c2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/6accc96c839a402d995cabc9677028c2 2023-04-25 06:55:07,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4266feff2a214a88b8840430fda4106a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4266feff2a214a88b8840430fda4106a 2023-04-25 06:55:07,907 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5b308ffd55b64041ad1fc611e227fd25 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5b308ffd55b64041ad1fc611e227fd25 2023-04-25 06:55:07,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4fdb83530ddf469ca7e7fc494300d832 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/4fdb83530ddf469ca7e7fc494300d832 2023-04-25 06:55:07,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/83762b99baa54b8bb4dbdae10122ea87 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/83762b99baa54b8bb4dbdae10122ea87 2023-04-25 06:55:07,910 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8816dc3a4ebf4c429145adb246b014e2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/8816dc3a4ebf4c429145adb246b014e2 2023-04-25 06:55:07,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/63ac5df342f0492c85a2d7fa97b37e4c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/63ac5df342f0492c85a2d7fa97b37e4c 2023-04-25 06:55:07,912 DEBUG [StoreCloser-TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5a1454e16ea8449ba0b8a93d224bbe52 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/5a1454e16ea8449ba0b8a93d224bbe52 2023-04-25 06:55:07,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/recovered.edits/828.seqid, newMaxSeqId=828, maxSeqId=1 2023-04-25 06:55:07,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17. 2023-04-25 06:55:07,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 11919fcb635bcaf9ea0c7782550d9d17: 2023-04-25 06:55:07,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:07,919 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=11919fcb635bcaf9ea0c7782550d9d17, regionState=CLOSED 2023-04-25 06:55:07,919 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405707919"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405707919"}]},"ts":"1682405707919"} 2023-04-25 06:55:07,921 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-25 06:55:07,921 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 11919fcb635bcaf9ea0c7782550d9d17, server=jenkins-hbase4.apache.org,41021,1682405622759 in 1.5100 sec 2023-04-25 06:55:07,922 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-25 06:55:07,922 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=11919fcb635bcaf9ea0c7782550d9d17, UNASSIGN in 1.5140 sec 2023-04-25 06:55:07,923 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405707923"}]},"ts":"1682405707923"} 2023-04-25 06:55:07,923 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 06:55:07,925 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 06:55:07,926 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5240 sec 2023-04-25 06:55:08,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 06:55:08,509 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-25 06:55:08,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 06:55:08,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:08,512 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:08,512 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:08,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-25 06:55:08,517 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:08,519 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/recovered.edits] 2023-04-25 06:55:08,521 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b6a0e996913423583012ff9069d38c0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/3b6a0e996913423583012ff9069d38c0 2023-04-25 06:55:08,522 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ad215947cb5440afadb22a9a5986417f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/A/ad215947cb5440afadb22a9a5986417f 2023-04-25 06:55:08,524 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/0228d5bbbab0451580166896d60c7a72 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/0228d5bbbab0451580166896d60c7a72 2023-04-25 06:55:08,525 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/db49e563d99342c28c24650ea4544419 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/B/db49e563d99342c28c24650ea4544419 2023-04-25 06:55:08,527 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/74a96bc19adb4308b2e31ddae6fa90de to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/74a96bc19adb4308b2e31ddae6fa90de 2023-04-25 06:55:08,528 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cbeef5c8e67a46bc98c27ee6077d2618 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/C/cbeef5c8e67a46bc98c27ee6077d2618 2023-04-25 06:55:08,531 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/recovered.edits/828.seqid to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17/recovered.edits/828.seqid 2023-04-25 06:55:08,531 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/11919fcb635bcaf9ea0c7782550d9d17 2023-04-25 06:55:08,531 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:55:08,533 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:08,534 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 06:55:08,536 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 06:55:08,537 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:08,537 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 06:55:08,537 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682405708537"}]},"ts":"9223372036854775807"} 2023-04-25 06:55:08,538 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 06:55:08,538 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 11919fcb635bcaf9ea0c7782550d9d17, NAME => 'TestAcidGuarantees,,1682405680635.11919fcb635bcaf9ea0c7782550d9d17.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 06:55:08,538 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 06:55:08,538 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682405708538"}]},"ts":"9223372036854775807"} 2023-04-25 06:55:08,540 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 06:55:08,542 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:08,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-25 06:55:08,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-25 06:55:08,614 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-25 06:55:08,625 INFO [Listener at localhost/41663] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 235), OpenFileDescriptor=547 (was 555), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=387 (was 442), ProcessCount=172 (was 172), AvailableMemoryMB=4257 (was 4301) 2023-04-25 06:55:08,636 INFO [Listener at localhost/41663] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=547, MaxFileDescriptor=60000, SystemLoadAverage=387, ProcessCount=172, AvailableMemoryMB=4257 2023-04-25 06:55:08,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:55:08,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:55:08,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:08,641 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 06:55:08,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-25 06:55:08,641 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 06:55:08,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 06:55:08,642 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:08,643 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e empty. 2023-04-25 06:55:08,643 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:08,643 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:55:08,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 06:55:08,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 06:55:09,050 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:55:09,052 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e275b2866a7bddedc0137edd1a42070e, NAME => 'TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp 2023-04-25 06:55:09,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 06:55:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:55:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e275b2866a7bddedc0137edd1a42070e, disabling compactions & flushes 2023-04-25 06:55:09,458 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. after waiting 0 ms 2023-04-25 06:55:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,458 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:09,460 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 06:55:09,461 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405709461"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405709461"}]},"ts":"1682405709461"} 2023-04-25 06:55:09,462 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 06:55:09,463 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 06:55:09,463 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405709463"}]},"ts":"1682405709463"} 2023-04-25 06:55:09,463 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 06:55:09,466 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, ASSIGN}] 2023-04-25 06:55:09,468 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, ASSIGN 2023-04-25 06:55:09,468 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=false 2023-04-25 06:55:09,619 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:09,619 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405709619"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405709619"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405709619"}]},"ts":"1682405709619"} 2023-04-25 06:55:09,621 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:55:09,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 06:55:09,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e275b2866a7bddedc0137edd1a42070e, NAME => 'TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:55:09,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:55:09,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,777 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,778 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A 2023-04-25 06:55:09,778 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A 2023-04-25 06:55:09,779 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:09,779 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e275b2866a7bddedc0137edd1a42070e columnFamilyName A 2023-04-25 06:55:09,779 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(310): Store=e275b2866a7bddedc0137edd1a42070e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:09,779 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,780 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B 2023-04-25 06:55:09,780 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B 2023-04-25 06:55:09,781 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:09,781 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e275b2866a7bddedc0137edd1a42070e columnFamilyName B 2023-04-25 06:55:09,781 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(310): Store=e275b2866a7bddedc0137edd1a42070e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:09,781 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,782 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C 2023-04-25 06:55:09,782 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C 2023-04-25 06:55:09,782 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:09,783 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e275b2866a7bddedc0137edd1a42070e columnFamilyName C 2023-04-25 06:55:09,783 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(310): Store=e275b2866a7bddedc0137edd1a42070e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:09,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:55:09,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:09,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:55:09,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e275b2866a7bddedc0137edd1a42070e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73095423, jitterRate=0.08920668065547943}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:55:09,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:09,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., pid=37, masterSystemTime=1682405709772 2023-04-25 06:55:09,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:09,790 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:09,790 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405709790"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405709790"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405709790"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405709790"}]},"ts":"1682405709790"} 2023-04-25 06:55:09,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-25 06:55:09,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 in 170 msec 2023-04-25 06:55:09,793 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-25 06:55:09,793 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, ASSIGN in 326 msec 2023-04-25 06:55:09,794 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 06:55:09,794 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405709794"}]},"ts":"1682405709794"} 2023-04-25 06:55:09,795 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 06:55:09,797 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 06:55:09,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-04-25 06:55:10,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 06:55:10,746 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-25 06:55:10,747 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x494529f9 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:10,752 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@118e131f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:10,753 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:10,755 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38606, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:10,756 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 06:55:10,757 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43198, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 06:55:10,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:55:10,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:55:10,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:11,169 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 06:55:11,170 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:55:11,170 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 06:55:11,172 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 06:55:11,174 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, REOPEN/MOVE}] 2023-04-25 06:55:11,174 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, REOPEN/MOVE 2023-04-25 06:55:11,175 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,175 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405711175"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405711175"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405711175"}]},"ts":"1682405711175"} 2023-04-25 06:55:11,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:55:11,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e275b2866a7bddedc0137edd1a42070e, disabling compactions & flushes 2023-04-25 06:55:11,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. after waiting 0 ms 2023-04-25 06:55:11,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 06:55:11,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:11,332 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e275b2866a7bddedc0137edd1a42070e to self. 2023-04-25 06:55:11,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,334 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=CLOSED 2023-04-25 06:55:11,334 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405711334"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405711334"}]},"ts":"1682405711334"} 2023-04-25 06:55:11,336 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-25 06:55:11,336 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 in 159 msec 2023-04-25 06:55:11,337 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=true 2023-04-25 06:55:11,487 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,487 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405711487"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405711487"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405711487"}]},"ts":"1682405711487"} 2023-04-25 06:55:11,489 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:55:11,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e275b2866a7bddedc0137edd1a42070e, NAME => 'TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:55:11,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:55:11,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,645 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,646 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A 2023-04-25 06:55:11,646 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A 2023-04-25 06:55:11,647 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:11,647 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e275b2866a7bddedc0137edd1a42070e columnFamilyName A 2023-04-25 06:55:11,648 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(310): Store=e275b2866a7bddedc0137edd1a42070e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:11,648 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,649 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B 2023-04-25 06:55:11,649 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B 2023-04-25 06:55:11,649 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:11,649 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e275b2866a7bddedc0137edd1a42070e columnFamilyName B 2023-04-25 06:55:11,650 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(310): Store=e275b2866a7bddedc0137edd1a42070e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:11,650 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,650 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C 2023-04-25 06:55:11,650 DEBUG [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C 2023-04-25 06:55:11,651 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:11,651 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e275b2866a7bddedc0137edd1a42070e columnFamilyName C 2023-04-25 06:55:11,651 INFO [StoreOpener-e275b2866a7bddedc0137edd1a42070e-1] regionserver.HStore(310): Store=e275b2866a7bddedc0137edd1a42070e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:11,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:55:11,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e275b2866a7bddedc0137edd1a42070e; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63774682, jitterRate=-0.049683183431625366}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:55:11,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:11,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., pid=42, masterSystemTime=1682405711640 2023-04-25 06:55:11,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,658 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,658 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,658 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405711658"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405711658"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405711658"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405711658"}]},"ts":"1682405711658"} 2023-04-25 06:55:11,660 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-25 06:55:11,660 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 in 170 msec 2023-04-25 06:55:11,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-25 06:55:11,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, REOPEN/MOVE in 486 msec 2023-04-25 06:55:11,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-25 06:55:11,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-25 06:55:11,664 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-04-25 06:55:11,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-25 06:55:11,665 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4d800352 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,669 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2deaf1fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,669 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x2329054b to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,673 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e7e18ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,673 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x30951255 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,676 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b7d4a51, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,676 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x5baa6768 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,681 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4335c1f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,681 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x63a535f3 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,684 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bbc9fc8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,685 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x678b4c3d to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,688 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6846dddb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,688 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x43736f95 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,692 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ea43962, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,692 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x72eaaf6e to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,698 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2abd7961, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,698 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4130667f to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,702 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@660630e8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,702 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x3b73a361 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:11,705 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@793239ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:11,714 DEBUG [hconnection-0x55eebd9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,715 DEBUG [hconnection-0x58863876-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:11,716 DEBUG [hconnection-0x72f19221-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,716 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,716 DEBUG [hconnection-0x3b1683db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,718 DEBUG [hconnection-0x1376e3fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,718 DEBUG [hconnection-0x14e1631d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,719 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,721 DEBUG [hconnection-0x1adeda5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,722 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38630, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,721 DEBUG [hconnection-0x3f634443-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,723 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38620, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,723 DEBUG [hconnection-0x733838d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,724 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,724 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:11,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:11,724 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:11,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:11,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:11,724 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:11,725 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,725 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:11,725 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:11,726 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:11,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:11,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:11,727 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,727 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:11,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:11,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:11,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:11,729 DEBUG [hconnection-0x6af5646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:11,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:11,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:11,730 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:11,730 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:11,731 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,737 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-25 06:55:11,737 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,738 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38688, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:11,739 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:11,739 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:11,739 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:11,739 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:11,739 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:11,739 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:11,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:11,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682405711746, max timestamp=1682405711753Num uniques 859; 2023-04-25 06:55:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 06:55:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682405711746, max timestamp=1682405711754Num uniques 859; 2023-04-25 06:55:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 06:55:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682405711746, max timestamp=1682405711753Num uniques 859; 2023-04-25 06:55:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 06:55:11,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1682405711758, max timestamp=1682405711767Num uniques 1268; 2023-04-25 06:55:11,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1682405711758, max timestamp=1682405711767Num uniques 1268; 2023-04-25 06:55:11,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=58068, new segment data size=19138, suffix heap size=159840,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1268, new segment cells count=418 2023-04-25 06:55:11,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1682405711758, max timestamp=1682405711767Num uniques 1268; 2023-04-25 06:55:11,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=58068, new segment data size=19138, suffix heap size=159840,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1268, new segment cells count=418 2023-04-25 06:55:11,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=58068, new segment data size=19138, suffix heap size=159840,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1268, new segment cells count=418 2023-04-25 06:55:11,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405771790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405771791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405771792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405771793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21832, totalHeapSize=49848, min timestamp=1682405711783, max timestamp=1682405711792Num uniques 1277; 2023-04-25 06:55:11,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=58472, new segment data size=21832, suffix heap size=160776,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1277, new segment cells count=477 2023-04-25 06:55:11,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21841, totalHeapSize=49848, min timestamp=1682405711783, max timestamp=1682405711792Num uniques 1277; 2023-04-25 06:55:11,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1682405711783, max timestamp=1682405711792Num uniques 1277; 2023-04-25 06:55:11,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=58481, new segment data size=21841, suffix heap size=160776,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1277, new segment cells count=477 2023-04-25 06:55:11,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=58482, new segment data size=21842, suffix heap size=160776,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1277, new segment cells count=477 2023-04-25 06:55:11,800 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251c4f466e584b43848cbba12e7f58a54a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251c4f466e584b43848cbba12e7f58a54a 2023-04-25 06:55:11,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405771815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405771894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405771895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405771895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405771895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:11,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:11,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405771916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405772100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405772100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405772100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405772100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405772121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,220 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=9, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/def1891eda3c49f79fae19274b57ee7c 2023-04-25 06:55:12,231 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/0014d1a4d16440638f66e025057bb2f2 2023-04-25 06:55:12,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405772405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405772405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405772409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405772409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405772426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,642 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/37ae7af6c0294b1fb11ab3178be45a9b 2023-04-25 06:55:12,646 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/def1891eda3c49f79fae19274b57ee7c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/def1891eda3c49f79fae19274b57ee7c 2023-04-25 06:55:12,650 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/def1891eda3c49f79fae19274b57ee7c, entries=100, sequenceid=9, filesize=18.4 K 2023-04-25 06:55:12,650 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/0014d1a4d16440638f66e025057bb2f2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/0014d1a4d16440638f66e025057bb2f2 2023-04-25 06:55:12,654 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/0014d1a4d16440638f66e025057bb2f2, entries=100, sequenceid=9, filesize=9.3 K 2023-04-25 06:55:12,655 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/37ae7af6c0294b1fb11ab3178be45a9b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/37ae7af6c0294b1fb11ab3178be45a9b 2023-04-25 06:55:12,658 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/37ae7af6c0294b1fb11ab3178be45a9b, entries=100, sequenceid=9, filesize=9.3 K 2023-04-25 06:55:12,659 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=174.43 KB/178620 for e275b2866a7bddedc0137edd1a42070e in 926ms, sequenceid=9, compaction requested=false 2023-04-25 06:55:12,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 06:55:12,660 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:12,660 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:12,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:12,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:12,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:12,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:12,666 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:12,667 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:12,667 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:12,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:12,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@681da952[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:12,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:12,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:12,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,670 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:12,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:12,674 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:12,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:12,676 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:12,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:12,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:12,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:12,676 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:12,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:12,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:12,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:12,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:12,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:12,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:12,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:12,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:12,688 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:12,688 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:12,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,788 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:12,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:12,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:12,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:12,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:12,792 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:12,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:12,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:12,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:12,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:12,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,796 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:12,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:12,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:12,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:12,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:12,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:12,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:12,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:12,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:12,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:12,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:12,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:12,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:12,798 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:12,798 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:12,798 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=174.43 KB heapSize=474.70 KB 2023-04-25 06:55:12,798 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:12,798 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:12,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:12,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:12,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:12,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:12,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:12,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405772914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405772914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405772915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405772915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:12,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:12,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405772936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405773019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405773019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,213 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d6bb5073e71548ef864a26da90c4551a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d6bb5073e71548ef864a26da90c4551a 2023-04-25 06:55:13,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405773222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405773222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,452 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 06:55:13,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405773526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405773527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,618 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=80, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/53e6a8872674490f8b3d04a327d5ea15 2023-04-25 06:55:13,638 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/d63471f3a654453fb0c0e827151e7133 2023-04-25 06:55:13,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405773926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405773928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:13,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405773948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:14,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405774031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:14,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405774033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,064 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/856a4c23294a494db14dd7931a9bf6f2 2023-04-25 06:55:14,074 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/53e6a8872674490f8b3d04a327d5ea15 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53e6a8872674490f8b3d04a327d5ea15 2023-04-25 06:55:14,078 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53e6a8872674490f8b3d04a327d5ea15, entries=150, sequenceid=80, filesize=25.1 K 2023-04-25 06:55:14,079 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/d63471f3a654453fb0c0e827151e7133 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/d63471f3a654453fb0c0e827151e7133 2023-04-25 06:55:14,086 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/d63471f3a654453fb0c0e827151e7133, entries=150, sequenceid=80, filesize=11.6 K 2023-04-25 06:55:14,087 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/856a4c23294a494db14dd7931a9bf6f2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/856a4c23294a494db14dd7931a9bf6f2 2023-04-25 06:55:14,091 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/856a4c23294a494db14dd7931a9bf6f2, entries=150, sequenceid=80, filesize=11.6 K 2023-04-25 06:55:14,091 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~474.66 KB/486048, currentSize=13.42 KB/13740 for e275b2866a7bddedc0137edd1a42070e in 1293ms, sequenceid=80, compaction requested=false 2023-04-25 06:55:14,091 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:14,091 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:14,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:14,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:14,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:14,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:14,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,093 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:14,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:14,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@889e82c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:14,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:14,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:14,098 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:14,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:14,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:14,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:14,103 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:14,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:14,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:14,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:14,103 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:14,105 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,105 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:14,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,105 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:14,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:14,105 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:14,107 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:14,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:14,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:14,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:14,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:14,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:14,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:14,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:14,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:14,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:14,109 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:14,109 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,110 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:14,110 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:14,110 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:14,110 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,210 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:14,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:14,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:14,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:14,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:14,214 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:14,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:14,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:14,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:14,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:14,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:14,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:14,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:14,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:14,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:14,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:14,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:14,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:14,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:14,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:14,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:14,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:14,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:14,219 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:14,219 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:14,219 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-25 06:55:14,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:14,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:14,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:14,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:14,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:14,220 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:14,248 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042555f7ec3ac7a842f29f1923a55313a063 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042555f7ec3ac7a842f29f1923a55313a063 2023-04-25 06:55:14,270 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/c7d515ee6f2541b99dbbe54320ca5f29 2023-04-25 06:55:14,694 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/fcad0a138b304a62a32096a16f3a10fd 2023-04-25 06:55:15,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:15,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:15,108 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/a590c13e4f8744458b6f8708aa31895c 2023-04-25 06:55:15,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405715087, max timestamp=1682405715108Num uniques 859; 2023-04-25 06:55:15,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:55:15,112 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/c7d515ee6f2541b99dbbe54320ca5f29 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c7d515ee6f2541b99dbbe54320ca5f29 2023-04-25 06:55:15,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405715087, max timestamp=1682405715108Num uniques 859; 2023-04-25 06:55:15,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:55:15,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405715087, max timestamp=1682405715108Num uniques 859; 2023-04-25 06:55:15,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:55:15,117 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c7d515ee6f2541b99dbbe54320ca5f29, entries=100, sequenceid=85, filesize=18.4 K 2023-04-25 06:55:15,118 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/fcad0a138b304a62a32096a16f3a10fd as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/fcad0a138b304a62a32096a16f3a10fd 2023-04-25 06:55:15,122 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/fcad0a138b304a62a32096a16f3a10fd, entries=100, sequenceid=85, filesize=9.3 K 2023-04-25 06:55:15,123 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/a590c13e4f8744458b6f8708aa31895c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a590c13e4f8744458b6f8708aa31895c 2023-04-25 06:55:15,128 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a590c13e4f8744458b6f8708aa31895c, entries=100, sequenceid=85, filesize=9.3 K 2023-04-25 06:55:15,129 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=67.09 KB/68700 for e275b2866a7bddedc0137edd1a42070e in 910ms, sequenceid=85, compaction requested=true 2023-04-25 06:55:15,129 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:15,129 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:15,131 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:15,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:15,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=73.80 KB heapSize=205.69 KB 2023-04-25 06:55:15,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:15,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:15,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:15,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:15,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:15,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:15,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749bdf42[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:15,137 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:15,137 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,138 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:15,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:15,138 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:15,139 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,139 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:15,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:15,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,140 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:15,140 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:15,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:15,146 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:15,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:15,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:15,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:15,146 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:15,151 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,151 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,151 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,152 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,153 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042564cb499521f249dfb3c3d3694b12c88f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042564cb499521f249dfb3c3d3694b12c88f 2023-04-25 06:55:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,166 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:15,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,167 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:15,167 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:15,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405775195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405775201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,267 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:15,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:15,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:15,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:15,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:15,271 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:15,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:15,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:15,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,272 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:15,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:15,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:15,274 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,275 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:15,275 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:15,275 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,276 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:15,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20436eec[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:15,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:15,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,282 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:15,283 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,283 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,283 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:15,286 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:15,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:15,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:15,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:15,286 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:15,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,288 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,288 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,288 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,295 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:15,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:15,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,296 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:15,296 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:15,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405775304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405775311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,396 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:15,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:15,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:15,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:15,400 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:15,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:15,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:15,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,401 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:15,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:15,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:15,403 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,403 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:15,403 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:15,403 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:15,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2591148f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:15,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:15,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,407 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:15,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:15,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:15,413 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:15,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:15,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:15,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:15,413 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:15,415 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,415 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,415 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,415 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,416 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,418 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:15,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:15,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:15,419 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:15,419 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:15,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,418 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405775510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,519 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:15,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:15,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:15,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:15,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:15,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,522 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:15,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405775517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,522 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,522 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:15,522 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:15,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,523 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,524 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,524 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,524 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,524 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:15,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:15,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:15,526 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,526 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:15,526 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:15,526 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:15,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,529 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:15,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@629e5099[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:15,531 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:15,531 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:15,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:15,533 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:15,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:15,536 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:15,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:15,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:15,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:15,536 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:15,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,538 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,539 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,541 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,541 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,541 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:15,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,542 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,541 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,542 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:15,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:15,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,560 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=113, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/bd63fe0e25754b7aa9fde16945403bda 2023-04-25 06:55:15,642 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:15,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:15,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:15,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:15,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:15,646 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:15,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:15,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:15,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,648 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,648 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:15,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:15,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:15,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:15,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:15,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:15,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:15,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:15,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:15,650 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:15,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:15,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4855e247[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:15,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:15,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:15,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:15,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:15,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,657 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:15,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:15,659 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:15,659 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:15,659 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:15,659 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:15,659 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:15,661 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,661 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,661 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:15,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:15,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:15,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,667 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,668 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:15,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,668 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:15,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,768 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:15,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:15,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:15,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:15,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:15,772 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:15,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:15,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:15,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,773 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:15,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:15,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:15,775 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,775 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:15,775 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:15,775 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:15,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,777 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:15,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73dad1af[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:15,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:15,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,782 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:15,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:15,782 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,782 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:15,782 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:15,784 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,784 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,784 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,784 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:15,787 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:15,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:15,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:15,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:15,787 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:15,788 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,788 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,788 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,789 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:15,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,792 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:15,792 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:15,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,793 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405775816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405775824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,893 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:15,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:15,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:15,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:15,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:15,896 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:15,896 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,896 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:15,896 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:15,897 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,897 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,899 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,899 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:15,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:15,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:15,902 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,902 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:15,902 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:15,902 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:15,903 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,903 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,903 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:15,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e429637[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:15,905 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:15,905 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:15,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:15,906 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:15,906 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,906 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:15,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:15,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:15,910 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:15,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:15,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:15,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:15,910 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:15,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:15,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:15,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:15,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:15,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:15,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:15,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:15,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:15,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:15,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,917 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,917 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:15,917 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:15,917 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:15,917 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:15,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405775938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,948 DEBUG [Thread-2222] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:15,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405775949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,957 DEBUG [Thread-2226] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:15,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:15,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405775957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:15,965 DEBUG [Thread-2228] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:15,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/3037885800814008ade22f55d1cfe374 2023-04-25 06:55:16,017 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:16,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:16,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:16,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:16,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:16,020 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:16,020 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,020 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:16,020 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:16,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:16,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:16,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:16,023 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:16,023 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,023 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:16,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:16,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:16,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:16,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:16,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:16,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:16,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:16,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:16,025 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e481ffe[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:16,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:16,027 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,029 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:16,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:16,031 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,031 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,031 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,031 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:16,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:16,033 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:16,033 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:16,033 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:16,033 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:16,033 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:16,035 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,035 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,035 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,035 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,037 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:16,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:16,039 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,039 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,039 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:16,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,140 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:16,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:16,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:16,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:16,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:16,143 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:16,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:16,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:16,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,143 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,149 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:16,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:16,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:16,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:16,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:16,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:16,152 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29e88baa[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:16,153 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:16,153 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,154 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,154 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:16,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:16,155 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:16,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,155 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:16,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:16,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:16,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:16,160 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:16,160 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:16,160 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:16,160 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:16,160 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:16,161 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,161 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,161 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,163 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,164 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,165 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,166 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:16,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:16,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:16,166 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,166 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:16,166 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:16,266 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:16,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:16,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:16,270 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:16,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:16,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:16,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,271 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,272 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,272 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:16,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:16,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:16,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:16,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9a6091[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:16,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:16,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:16,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:16,278 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,278 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:16,282 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:16,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:16,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:16,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:16,282 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:16,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,285 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,285 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,287 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:16,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,289 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:16,289 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:16,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:16,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405776326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:16,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405776330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/8395b33ff423440fbdf68c29cc963460 2023-04-25 06:55:16,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/bd63fe0e25754b7aa9fde16945403bda as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/bd63fe0e25754b7aa9fde16945403bda 2023-04-25 06:55:16,389 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:16,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:16,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/bd63fe0e25754b7aa9fde16945403bda, entries=200, sequenceid=113, filesize=31.9 K 2023-04-25 06:55:16,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:16,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/3037885800814008ade22f55d1cfe374 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/3037885800814008ade22f55d1cfe374 2023-04-25 06:55:16,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:16,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:16,394 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:16,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:16,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:16,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,395 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:16,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:16,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:16,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/3037885800814008ade22f55d1cfe374, entries=150, sequenceid=113, filesize=11.6 K 2023-04-25 06:55:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:16,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/8395b33ff423440fbdf68c29cc963460 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8395b33ff423440fbdf68c29cc963460 2023-04-25 06:55:16,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,400 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,400 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8395b33ff423440fbdf68c29cc963460, entries=150, sequenceid=113, filesize=11.6 K 2023-04-25 06:55:16,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~205.64 KB/210576, currentSize=107.34 KB/109920 for e275b2866a7bddedc0137edd1a42070e in 1269ms, sequenceid=113, compaction requested=true 2023-04-25 06:55:16,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:16,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:16,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:16,402 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:16,402 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:16,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:55:16,404 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 96028 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,404 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/A is initiating minor compaction (all files) 2023-04-25 06:55:16,404 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/A in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,404 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/def1891eda3c49f79fae19274b57ee7c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53e6a8872674490f8b3d04a327d5ea15, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c7d515ee6f2541b99dbbe54320ca5f29, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/bd63fe0e25754b7aa9fde16945403bda] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=93.8 K 2023-04-25 06:55:16,404 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:16,404 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/B is initiating minor compaction (all files) 2023-04-25 06:55:16,404 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting def1891eda3c49f79fae19274b57ee7c, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682405711725 2023-04-25 06:55:16,404 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/B in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,405 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/0014d1a4d16440638f66e025057bb2f2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/d63471f3a654453fb0c0e827151e7133, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/fcad0a138b304a62a32096a16f3a10fd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/3037885800814008ade22f55d1cfe374] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=41.9 K 2023-04-25 06:55:16,405 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 53e6a8872674490f8b3d04a327d5ea15, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682405711809 2023-04-25 06:55:16,405 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 0014d1a4d16440638f66e025057bb2f2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682405711725 2023-04-25 06:55:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,406 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c7d515ee6f2541b99dbbe54320ca5f29, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405712912 2023-04-25 06:55:16,406 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d63471f3a654453fb0c0e827151e7133, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682405711809 2023-04-25 06:55:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:16,406 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting bd63fe0e25754b7aa9fde16945403bda, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682405715122 2023-04-25 06:55:16,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:16,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4623ac1b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:16,406 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting fcad0a138b304a62a32096a16f3a10fd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405712912 2023-04-25 06:55:16,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,407 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 3037885800814008ade22f55d1cfe374, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682405715125 2023-04-25 06:55:16,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,408 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:16,408 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:16,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:16,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:16,414 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:16,414 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:16,414 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:16,414 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:16,414 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:16,415 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,415 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:16,415 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:16,416 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,418 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,423 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:16,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:16,423 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,423 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:16,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,423 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:16,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:16,424 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:16,423 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:16,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:16,429 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#B#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:16,431 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#A#compaction#297 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:16,524 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:16,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:16,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:16,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:16,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:16,527 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:16,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:16,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:16,528 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,528 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:16,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:16,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:16,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:16,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:16,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:16,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:16,530 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:16,530 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:16,530 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:16,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:16,531 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:16,531 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,531 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:16,531 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:55:16,531 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:16,531 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:16,844 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/7bd8d2779c2d45219c4fcc4f177e537a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/7bd8d2779c2d45219c4fcc4f177e537a 2023-04-25 06:55:16,848 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/B of e275b2866a7bddedc0137edd1a42070e into 7bd8d2779c2d45219c4fcc4f177e537a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:16,848 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:16,848 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/B, priority=12, startTime=1682405716402; duration=0sec 2023-04-25 06:55:16,848 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:16,848 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:16,850 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:16,850 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/C is initiating minor compaction (all files) 2023-04-25 06:55:16,850 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/C in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:16,850 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/37ae7af6c0294b1fb11ab3178be45a9b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/856a4c23294a494db14dd7931a9bf6f2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a590c13e4f8744458b6f8708aa31895c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8395b33ff423440fbdf68c29cc963460] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=41.9 K 2023-04-25 06:55:16,851 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 37ae7af6c0294b1fb11ab3178be45a9b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682405711725 2023-04-25 06:55:16,852 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 856a4c23294a494db14dd7931a9bf6f2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682405711809 2023-04-25 06:55:16,852 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a590c13e4f8744458b6f8708aa31895c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682405712912 2023-04-25 06:55:16,852 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 8395b33ff423440fbdf68c29cc963460, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682405715125 2023-04-25 06:55:16,860 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#C#compaction#300 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:16,949 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ee514062ddb34e9c80a8624de6617169 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ee514062ddb34e9c80a8624de6617169 2023-04-25 06:55:17,267 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/63857662ec68439daa2127bd11a0a75c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63857662ec68439daa2127bd11a0a75c 2023-04-25 06:55:17,272 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/2aada0af044549f7a227edab4152793d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/2aada0af044549f7a227edab4152793d 2023-04-25 06:55:17,274 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/A of e275b2866a7bddedc0137edd1a42070e into 63857662ec68439daa2127bd11a0a75c(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:17,275 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:17,275 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/A, priority=12, startTime=1682405716402; duration=0sec 2023-04-25 06:55:17,275 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:17,279 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/C of e275b2866a7bddedc0137edd1a42070e into 2aada0af044549f7a227edab4152793d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:17,279 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:17,279 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/C, priority=12, startTime=1682405716402; duration=0sec 2023-04-25 06:55:17,279 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:17,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:17,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:17,354 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=132, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/b3e4895e5390443a899f0ed105b90634 2023-04-25 06:55:17,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405777361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405777366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405777468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:17,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405777468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:17,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405777669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:17,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405777670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,768 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/429aa264a2b24f218fd42853bb6be71f 2023-04-25 06:55:17,778 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/4b8ba5f9331c4266ab4715cefcaf345b 2023-04-25 06:55:17,782 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/b3e4895e5390443a899f0ed105b90634 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/b3e4895e5390443a899f0ed105b90634 2023-04-25 06:55:17,786 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/b3e4895e5390443a899f0ed105b90634, entries=150, sequenceid=132, filesize=25.2 K 2023-04-25 06:55:17,787 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/429aa264a2b24f218fd42853bb6be71f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/429aa264a2b24f218fd42853bb6be71f 2023-04-25 06:55:17,792 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/429aa264a2b24f218fd42853bb6be71f, entries=150, sequenceid=132, filesize=11.7 K 2023-04-25 06:55:17,793 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/4b8ba5f9331c4266ab4715cefcaf345b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4b8ba5f9331c4266ab4715cefcaf345b 2023-04-25 06:55:17,801 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4b8ba5f9331c4266ab4715cefcaf345b, entries=150, sequenceid=132, filesize=11.7 K 2023-04-25 06:55:17,802 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e275b2866a7bddedc0137edd1a42070e in 1271ms, sequenceid=132, compaction requested=false 2023-04-25 06:55:17,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:17,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:17,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,806 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@489aecce[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:17,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:17,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:17,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:17,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:17,816 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:17,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:17,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:17,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:17,816 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:17,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:17,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:17,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:17,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:17,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:17,827 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:17,827 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:17,827 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:17,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:17,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:17,928 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:17,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:17,931 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:17,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:17,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:17,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:17,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:17,933 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:17,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:17,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:17,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:17,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:17,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:17,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:17,936 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:17,948 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c50bf27733f64e6fbf06205ed28e0dfd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c50bf27733f64e6fbf06205ed28e0dfd 2023-04-25 06:55:17,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:17,976 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:18,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405778014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405778015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405778116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405778125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405778321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405778329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,363 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/c16c9e8619674915af62b393459ebff9 2023-04-25 06:55:18,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405778627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:18,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405778634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:18,773 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/992d8ce99cc5460392e957b0611b51a0 2023-04-25 06:55:19,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:19,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405779135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:19,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405779142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,183 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/f465cff9d9274e67ad505a488a1bd1f7 2023-04-25 06:55:19,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/c16c9e8619674915af62b393459ebff9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c16c9e8619674915af62b393459ebff9 2023-04-25 06:55:19,192 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c16c9e8619674915af62b393459ebff9, entries=150, sequenceid=148, filesize=25.3 K 2023-04-25 06:55:19,193 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/992d8ce99cc5460392e957b0611b51a0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/992d8ce99cc5460392e957b0611b51a0 2023-04-25 06:55:19,199 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/992d8ce99cc5460392e957b0611b51a0, entries=150, sequenceid=148, filesize=11.8 K 2023-04-25 06:55:19,200 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/f465cff9d9274e67ad505a488a1bd1f7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f465cff9d9274e67ad505a488a1bd1f7 2023-04-25 06:55:19,205 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f465cff9d9274e67ad505a488a1bd1f7, entries=150, sequenceid=148, filesize=11.8 K 2023-04-25 06:55:19,206 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e275b2866a7bddedc0137edd1a42070e in 1270ms, sequenceid=148, compaction requested=true 2023-04-25 06:55:19,206 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:19,206 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:19,207 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25c29bac[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:19,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:19,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,211 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:19,212 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,212 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:19,213 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:19,215 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:19,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:19,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:19,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:19,215 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:19,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:19,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:19,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:19,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:19,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:19,221 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:19,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,221 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:19,221 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:19,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:19,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:19,321 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:19,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:19,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:19,325 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:19,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:19,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:19,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:19,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,327 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:19,327 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:19,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:19,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:19,329 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:19,329 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:19,329 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:55:19,330 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:19,330 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:19,330 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:19,330 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:19,330 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:19,330 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:19,344 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425fbc9bd12bb914600b8713c834930ad7c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425fbc9bd12bb914600b8713c834930ad7c 2023-04-25 06:55:19,349 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/5f7440b0ae0a4409a103b83d2e96b555 2023-04-25 06:55:19,363 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/b734aaa5ba7d48998e6162f6b796e6a0 2023-04-25 06:55:19,774 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/511d748cc2d14ed3bb3d0832e6339f40 2023-04-25 06:55:19,778 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/5f7440b0ae0a4409a103b83d2e96b555 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/5f7440b0ae0a4409a103b83d2e96b555 2023-04-25 06:55:19,781 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/5f7440b0ae0a4409a103b83d2e96b555, entries=150, sequenceid=167, filesize=25.3 K 2023-04-25 06:55:19,782 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/b734aaa5ba7d48998e6162f6b796e6a0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/b734aaa5ba7d48998e6162f6b796e6a0 2023-04-25 06:55:19,786 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/b734aaa5ba7d48998e6162f6b796e6a0, entries=150, sequenceid=167, filesize=11.8 K 2023-04-25 06:55:19,790 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/511d748cc2d14ed3bb3d0832e6339f40 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/511d748cc2d14ed3bb3d0832e6339f40 2023-04-25 06:55:19,794 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/511d748cc2d14ed3bb3d0832e6339f40, entries=150, sequenceid=167, filesize=11.8 K 2023-04-25 06:55:19,795 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for e275b2866a7bddedc0137edd1a42070e in 465ms, sequenceid=167, compaction requested=true 2023-04-25 06:55:19,795 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:19,795 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:19,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a03e96d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:19,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:19,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,807 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,807 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:19,807 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:19,807 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:19,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:19,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:19,812 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:19,812 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:19,812 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:19,812 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:19,812 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:19,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:19,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:19,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:19,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:19,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:19,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:19,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:19,821 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:19,821 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:19,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:19,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:19,921 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:19,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:19,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:19,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:19,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:19,925 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:19,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:19,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:19,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:19,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,926 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:19,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:19,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:19,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:19,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:19,928 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:19,928 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:19,928 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:19,928 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:19,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:19,930 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@219bd142[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:19,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:19,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,935 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:19,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:19,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,938 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:19,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:19,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:19,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:19,941 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:19,941 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:19,941 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:19,941 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:19,941 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:19,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:19,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:19,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:19,945 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:19,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:19,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:19,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:19,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:19,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:19,947 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:19,947 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:19,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:19,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,047 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:20,050 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:55:20,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:20,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:20,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:20,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:20,051 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,051 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,051 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,051 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,054 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,055 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,055 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,056 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,056 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,058 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e78a523[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,063 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,063 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,063 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,063 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,063 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,064 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,065 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,068 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,068 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,071 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,071 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,071 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,073 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:20,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,076 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,076 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405720084, max timestamp=1682405720106Num uniques 859; 2023-04-25 06:55:20,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:20,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405720084, max timestamp=1682405720106Num uniques 859; 2023-04-25 06:55:20,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405720084, max timestamp=1682405720106Num uniques 859; 2023-04-25 06:55:20,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:20,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:20,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405780142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405780143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405780144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405780147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405780147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,176 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,179 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,179 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,180 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,180 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,180 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,180 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,181 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,181 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,181 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,181 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,183 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,183 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,183 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,183 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,184 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5656bb[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,192 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,192 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,193 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,193 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,194 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,194 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,196 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,196 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,198 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,199 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,202 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,202 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,203 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,203 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,203 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,203 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,203 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,203 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,203 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,203 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,203 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:20,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405780247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405780253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405780253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,303 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,306 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,309 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:20,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,311 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,311 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,312 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,312 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a7b8ebd[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,315 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,315 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,316 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,320 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,320 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,320 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,320 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,320 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,325 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,326 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,330 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,331 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:20,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,331 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,431 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,434 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,434 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,434 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,435 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,435 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,435 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,436 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,436 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,436 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,437 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:20,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,438 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,438 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,438 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,438 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,440 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15154371[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,443 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,445 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,447 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,447 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,447 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,447 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,447 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,449 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,449 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,449 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,449 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,452 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,453 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,453 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405780459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405780462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405780462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,489 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252b3859de0a1b4b6cad934ad6e19ca1b6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252b3859de0a1b4b6cad934ad6e19ca1b6 2023-04-25 06:55:20,553 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,556 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,559 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:20,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,561 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,561 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,561 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,561 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e3fbd4d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,565 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,566 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,566 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,567 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,569 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,569 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,569 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,569 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,569 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,573 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,574 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,576 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,576 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,576 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,576 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,576 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,576 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,576 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:20,576 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,576 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,577 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,576 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,577 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,677 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,680 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,681 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:20,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,683 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,684 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e1cdbf[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,688 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,688 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,691 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,692 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,692 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,692 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,693 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,693 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,693 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,695 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,698 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,699 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,699 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405780764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405780765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:20,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405780766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,799 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,802 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:20,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,806 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,807 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d9bf18[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,810 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,811 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,811 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,816 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,816 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:20,822 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,822 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,895 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=178, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/21bcec1ae4a44a178b7f33cd4067ef40 2023-04-25 06:55:20,922 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:20,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:20,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:20,925 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:20,925 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:20,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:20,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,928 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,928 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:20,928 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:20,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:20,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:20,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:20,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:20,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:20,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:20,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:20,929 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,930 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:20,930 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:20,930 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:20,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:20,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:20,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:20,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:20,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,932 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:20,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae86619[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:20,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:20,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,935 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:20,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:20,936 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:20,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:20,939 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:20,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:20,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:20,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:20,939 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:20,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,940 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:20,941 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:20,942 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,945 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,945 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:20,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:20,945 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:20,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:20,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:20,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:20,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:20,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:20,946 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:20,946 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:20,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:20,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:20,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:20,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,046 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:21,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:21,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:21,050 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:21,050 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,050 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:21,050 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:21,050 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,051 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:21,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:21,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:21,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:21,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:21,053 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,054 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:21,054 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:21,054 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:21,055 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,055 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:21,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492e7ddc[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:21,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:21,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,058 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:21,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:21,059 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:21,059 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,059 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:21,061 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,061 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,061 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:21,061 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:21,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:21,063 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:21,063 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:21,063 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:21,063 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:21,063 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:21,065 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,065 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,065 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:21,065 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,067 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:21,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:21,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:21,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:21,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:21,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:21,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:21,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,070 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:21,070 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:21,170 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:21,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:21,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:21,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:21,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:21,173 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:21,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:21,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:21,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,178 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,178 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:21,179 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,179 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:21,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:21,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:21,181 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,181 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:21,181 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:21,182 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:21,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@147fdd73[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:21,184 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:21,184 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,185 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:21,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:21,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:21,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:21,188 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:21,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:21,190 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:21,190 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:21,190 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:21,190 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:21,190 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:21,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,192 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:21,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,192 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,193 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,195 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:21,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:21,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:21,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:21,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:21,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:21,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:21,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:21,196 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:21,196 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:21,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:21,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405781273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:21,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405781277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:21,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405781277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,296 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:21,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:21,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:21,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:21,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:21,300 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:21,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:21,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:21,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,301 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:21,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:21,302 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,302 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:21,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:21,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:21,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:21,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:21,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:21,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/19655ab151774980815b78431acafcad 2023-04-25 06:55:21,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,306 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c76819b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:21,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:21,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,310 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:21,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:21,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/c5ef0798df014eb99e76c1ccecf6d1db 2023-04-25 06:55:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:21,321 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/21bcec1ae4a44a178b7f33cd4067ef40 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/21bcec1ae4a44a178b7f33cd4067ef40 2023-04-25 06:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:21,325 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:21,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:21,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:21,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:21,325 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:21,328 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,328 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:21,328 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/21bcec1ae4a44a178b7f33cd4067ef40, entries=350, sequenceid=178, filesize=52.6 K 2023-04-25 06:55:21,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/19655ab151774980815b78431acafcad as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/19655ab151774980815b78431acafcad 2023-04-25 06:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:21,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:21,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/19655ab151774980815b78431acafcad, entries=150, sequenceid=178, filesize=11.8 K 2023-04-25 06:55:21,334 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:21,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/c5ef0798df014eb99e76c1ccecf6d1db as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/c5ef0798df014eb99e76c1ccecf6d1db 2023-04-25 06:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:21,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:21,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:21,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:21,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:21,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:21,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:21,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:21,337 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:21,337 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:21,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:21,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/c5ef0798df014eb99e76c1ccecf6d1db, entries=150, sequenceid=178, filesize=11.8 K 2023-04-25 06:55:21,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e275b2866a7bddedc0137edd1a42070e in 1289ms, sequenceid=178, compaction requested=true 2023-04-25 06:55:21,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:21,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:21,339 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:55:21,340 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:55:21,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:21,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:21,341 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 157382 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:55:21,341 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60118 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:55:21,341 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/A is initiating minor compaction (all files) 2023-04-25 06:55:21,341 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/B is initiating minor compaction (all files) 2023-04-25 06:55:21,341 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/A in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,341 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/B in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,341 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63857662ec68439daa2127bd11a0a75c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/b3e4895e5390443a899f0ed105b90634, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c16c9e8619674915af62b393459ebff9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/5f7440b0ae0a4409a103b83d2e96b555, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/21bcec1ae4a44a178b7f33cd4067ef40] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=153.7 K 2023-04-25 06:55:21,341 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/7bd8d2779c2d45219c4fcc4f177e537a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/429aa264a2b24f218fd42853bb6be71f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/992d8ce99cc5460392e957b0611b51a0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/b734aaa5ba7d48998e6162f6b796e6a0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/19655ab151774980815b78431acafcad] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=58.7 K 2023-04-25 06:55:21,342 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 63857662ec68439daa2127bd11a0a75c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682405715125 2023-04-25 06:55:21,342 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7bd8d2779c2d45219c4fcc4f177e537a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682405715125 2023-04-25 06:55:21,342 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b3e4895e5390443a899f0ed105b90634, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682405715179 2023-04-25 06:55:21,342 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 429aa264a2b24f218fd42853bb6be71f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682405715179 2023-04-25 06:55:21,343 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c16c9e8619674915af62b393459ebff9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682405717346 2023-04-25 06:55:21,343 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 992d8ce99cc5460392e957b0611b51a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682405717346 2023-04-25 06:55:21,343 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5f7440b0ae0a4409a103b83d2e96b555, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682405718006 2023-04-25 06:55:21,343 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b734aaa5ba7d48998e6162f6b796e6a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682405718006 2023-04-25 06:55:21,344 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 21bcec1ae4a44a178b7f33cd4067ef40, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682405720020 2023-04-25 06:55:21,344 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 19655ab151774980815b78431acafcad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682405720020 2023-04-25 06:55:21,355 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#B#compaction#312 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:21,362 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#A#compaction#313 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:21,437 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:21,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:21,441 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:21,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:21,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:21,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,442 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:21,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:21,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:21,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:21,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:21,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:21,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:21,444 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,444 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:21,444 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-25 06:55:21,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:21,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:21,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:21,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:21,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:21,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:21,458 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f0cedf9283d447a1899197d761d69422 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f0cedf9283d447a1899197d761d69422 2023-04-25 06:55:21,764 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/33ca28ca442e4830a7fcc3d0f16e6e45 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/33ca28ca442e4830a7fcc3d0f16e6e45 2023-04-25 06:55:21,768 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/B of e275b2866a7bddedc0137edd1a42070e into 33ca28ca442e4830a7fcc3d0f16e6e45(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:21,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:21,768 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/B, priority=11, startTime=1682405721340; duration=0sec 2023-04-25 06:55:21,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:21,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:55:21,769 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60118 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:55:21,770 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/C is initiating minor compaction (all files) 2023-04-25 06:55:21,770 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/C in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:21,770 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/2aada0af044549f7a227edab4152793d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4b8ba5f9331c4266ab4715cefcaf345b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f465cff9d9274e67ad505a488a1bd1f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/511d748cc2d14ed3bb3d0832e6339f40, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/c5ef0798df014eb99e76c1ccecf6d1db] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=58.7 K 2023-04-25 06:55:21,770 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2aada0af044549f7a227edab4152793d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1682405715125 2023-04-25 06:55:21,771 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4b8ba5f9331c4266ab4715cefcaf345b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682405715179 2023-04-25 06:55:21,771 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting f465cff9d9274e67ad505a488a1bd1f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682405717346 2023-04-25 06:55:21,772 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 511d748cc2d14ed3bb3d0832e6339f40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682405718006 2023-04-25 06:55:21,772 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c5ef0798df014eb99e76c1ccecf6d1db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682405720020 2023-04-25 06:55:21,789 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#C#compaction#315 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:21,868 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/2f7d3513be7f41458c537f5eeb65f0a3 2023-04-25 06:55:22,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:22,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:22,185 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/57b9e83b702e4ac3b2f5d169b6c85a8f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/57b9e83b702e4ac3b2f5d169b6c85a8f 2023-04-25 06:55:22,189 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/A of e275b2866a7bddedc0137edd1a42070e into 57b9e83b702e4ac3b2f5d169b6c85a8f(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:22,189 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:22,189 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/A, priority=11, startTime=1682405721339; duration=0sec 2023-04-25 06:55:22,189 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:22,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405782189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,198 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/d2e48f04891a4345898fafc191dd0758 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d2e48f04891a4345898fafc191dd0758 2023-04-25 06:55:22,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405782190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,202 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/C of e275b2866a7bddedc0137edd1a42070e into d2e48f04891a4345898fafc191dd0758(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:22,202 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:22,202 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/C, priority=11, startTime=1682405721340; duration=0sec 2023-04-25 06:55:22,202 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:22,279 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/c1013d3e67724a24b94267febd45be0f 2023-04-25 06:55:22,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405782283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,291 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/87b272cb4ebd4b858658bb50f9d4c436 2023-04-25 06:55:22,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405782291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405782292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,297 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/2f7d3513be7f41458c537f5eeb65f0a3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2f7d3513be7f41458c537f5eeb65f0a3 2023-04-25 06:55:22,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405782294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,301 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2f7d3513be7f41458c537f5eeb65f0a3, entries=150, sequenceid=214, filesize=25.3 K 2023-04-25 06:55:22,301 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/c1013d3e67724a24b94267febd45be0f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/c1013d3e67724a24b94267febd45be0f 2023-04-25 06:55:22,305 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/c1013d3e67724a24b94267febd45be0f, entries=150, sequenceid=214, filesize=11.8 K 2023-04-25 06:55:22,306 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/87b272cb4ebd4b858658bb50f9d4c436 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/87b272cb4ebd4b858658bb50f9d4c436 2023-04-25 06:55:22,309 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/87b272cb4ebd4b858658bb50f9d4c436, entries=150, sequenceid=214, filesize=11.8 K 2023-04-25 06:55:22,310 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e275b2866a7bddedc0137edd1a42070e in 866ms, sequenceid=214, compaction requested=false 2023-04-25 06:55:22,310 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:22,310 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:22,312 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:22,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2537f0a2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:22,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:22,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,315 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:22,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:22,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:22,316 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,321 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:22,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:22,328 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:22,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:22,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:22,328 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:22,328 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:22,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:22,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 06:55:22,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:22,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:22,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:22,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:22,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:22,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:22,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,332 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,333 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,335 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:22,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:22,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,336 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,336 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:22,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,358 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425910e49e697994e0799244499dfc157b3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425910e49e697994e0799244499dfc157b3 2023-04-25 06:55:22,436 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:22,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:22,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:22,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:22,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:22,444 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:22,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:22,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:22,445 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,445 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,447 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,447 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,447 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,447 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:22,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:22,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:22,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:22,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:22,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:22,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:22,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:22,450 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,450 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:22,450 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:22,450 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:22,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4db945[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:22,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:22,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:22,459 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,459 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:22,461 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:22,463 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:22,463 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:22,463 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:22,463 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:22,463 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:22,464 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,464 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,470 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,470 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:22,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:22,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,471 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:22,471 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:22,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405722509, max timestamp=1682405722543Num uniques 859; 2023-04-25 06:55:22,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405722509, max timestamp=1682405722543Num uniques 859; 2023-04-25 06:55:22,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405722509, max timestamp=1682405722543Num uniques 859; 2023-04-25 06:55:22,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:22,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:22,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:22,572 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:22,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:22,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:22,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:22,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:22,575 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:22,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:22,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:22,577 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,577 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:22,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:22,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:22,580 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,580 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:22,580 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:22,580 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:22,582 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:22,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75336b9a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:22,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:22,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,584 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:22,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:22,587 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:22,589 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:22,589 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:22,589 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:22,589 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:22,589 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:22,590 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,590 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,590 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,591 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,593 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,594 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,594 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:22,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:22,595 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,595 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:22,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,595 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,595 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405782599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405782603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,695 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:22,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:22,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:22,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:22,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:22,699 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:22,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:22,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:22,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,700 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,700 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:22,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:22,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:22,702 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,702 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:22,702 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:22,702 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:22,703 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:22,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,705 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,705 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,705 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:22,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc156fe[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:22,707 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:22,707 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,708 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:22,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:22,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:22,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:22,709 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:22,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:22,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:22,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:22,711 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:22,711 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:22,711 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:22,712 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:22,712 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:22,713 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,713 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,713 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,713 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,715 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405782710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405782714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:22,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:22,719 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,718 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,719 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:22,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,719 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,719 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,763 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=228, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/63cde6bab0e141da8f1392d940a8ef9d 2023-04-25 06:55:22,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/e2de162ad4a542e4901b380eae9c0ebc 2023-04-25 06:55:22,819 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:22,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:22,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:22,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:22,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:22,823 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:22,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:22,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:22,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:22,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:22,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:22,827 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,827 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:22,827 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:22,827 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:22,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:22,830 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:22,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67690b04[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:22,831 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:22,831 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,832 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:22,833 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,833 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:22,837 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:22,839 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:22,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:22,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:22,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:22,840 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:22,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,843 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:22,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:22,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,845 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:22,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,846 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:22,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,846 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:22,846 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682405782918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:22,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405782920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,946 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:22,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:22,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:22,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:22,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:22,949 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:22,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:22,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:22,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,951 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,951 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:22,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:22,953 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,953 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:22,953 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:22,953 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:22,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,955 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159c113d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:22,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:22,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,958 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:22,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,960 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:22,962 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:22,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:22,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:22,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:22,963 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:22,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:22,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:22,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:22,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:22,970 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:22,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:22,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:22,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:22,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:22,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:22,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:22,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:22,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:22,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:22,970 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:22,970 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:23,070 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:23,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:23,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:23,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:23,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:23,074 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:23,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:23,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:23,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:23,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:23,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:23,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:23,079 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9dc7af[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:23,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:23,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:23,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:23,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:23,087 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:23,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:23,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:23,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:23,087 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:23,088 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,088 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,088 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:23,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:23,092 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:23,093 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:23,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/6111d21e29ac459bb1945cc9bdb178d5 2023-04-25 06:55:23,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/63cde6bab0e141da8f1392d940a8ef9d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63cde6bab0e141da8f1392d940a8ef9d 2023-04-25 06:55:23,193 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:23,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:23,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63cde6bab0e141da8f1392d940a8ef9d, entries=150, sequenceid=228, filesize=25.3 K 2023-04-25 06:55:23,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:23,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/e2de162ad4a542e4901b380eae9c0ebc as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/e2de162ad4a542e4901b380eae9c0ebc 2023-04-25 06:55:23,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:23,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:23,196 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:23,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:23,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:23,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,196 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,197 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:23,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:23,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:23,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:23,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/e2de162ad4a542e4901b380eae9c0ebc, entries=100, sequenceid=228, filesize=9.4 K 2023-04-25 06:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:23,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/6111d21e29ac459bb1945cc9bdb178d5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/6111d21e29ac459bb1945cc9bdb178d5 2023-04-25 06:55:23,202 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/6111d21e29ac459bb1945cc9bdb178d5, entries=100, sequenceid=228, filesize=9.4 K 2023-04-25 06:55:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e275b2866a7bddedc0137edd1a42070e in 874ms, sequenceid=228, compaction requested=true 2023-04-25 06:55:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:23,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:23,205 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:23,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:23,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:23,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:55:23,205 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:23,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:23,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9c1884[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:23,206 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:23,206 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77992 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:23,206 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/B is initiating minor compaction (all files) 2023-04-25 06:55:23,206 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/A is initiating minor compaction (all files) 2023-04-25 06:55:23,206 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/B in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,206 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/A in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,206 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/33ca28ca442e4830a7fcc3d0f16e6e45, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/c1013d3e67724a24b94267febd45be0f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/e2de162ad4a542e4901b380eae9c0ebc] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=33.2 K 2023-04-25 06:55:23,206 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/57b9e83b702e4ac3b2f5d169b6c85a8f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2f7d3513be7f41458c537f5eeb65f0a3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63cde6bab0e141da8f1392d940a8ef9d] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=76.2 K 2023-04-25 06:55:23,206 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:23,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:23,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:23,207 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,207 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:23,207 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 57b9e83b702e4ac3b2f5d169b6c85a8f, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682405720020 2023-04-25 06:55:23,207 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 33ca28ca442e4830a7fcc3d0f16e6e45, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682405720020 2023-04-25 06:55:23,208 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c1013d3e67724a24b94267febd45be0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682405720141 2023-04-25 06:55:23,208 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 2f7d3513be7f41458c537f5eeb65f0a3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682405720141 2023-04-25 06:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,208 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,208 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 63cde6bab0e141da8f1392d940a8ef9d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405722188 2023-04-25 06:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:23,208 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e2de162ad4a542e4901b380eae9c0ebc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405722188 2023-04-25 06:55:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:23,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:23,216 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:23,216 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:23,216 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:23,216 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:23,216 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:23,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:23,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:23,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,227 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:23,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:23,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-25 06:55:23,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,227 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:23,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:23,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:23,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:23,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:23,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:23,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:23,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,233 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#B#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:23,265 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#A#compaction#322 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:23,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405783256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405783260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,271 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425402aa40c0bf544279e4ad7d032b9e2a8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425402aa40c0bf544279e4ad7d032b9e2a8 2023-04-25 06:55:23,312 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/aaba3cbad9b3497fa4f6c5755e2367b4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/aaba3cbad9b3497fa4f6c5755e2367b4 2023-04-25 06:55:23,316 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/A of e275b2866a7bddedc0137edd1a42070e into aaba3cbad9b3497fa4f6c5755e2367b4(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:23,316 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:23,316 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/A, priority=13, startTime=1682405723205; duration=0sec 2023-04-25 06:55:23,316 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:23,316 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:23,317 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:23,317 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/C is initiating minor compaction (all files) 2023-04-25 06:55:23,317 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/C in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,317 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d2e48f04891a4345898fafc191dd0758, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/87b272cb4ebd4b858658bb50f9d4c436, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/6111d21e29ac459bb1945cc9bdb178d5] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=33.2 K 2023-04-25 06:55:23,317 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d2e48f04891a4345898fafc191dd0758, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682405720020 2023-04-25 06:55:23,318 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 87b272cb4ebd4b858658bb50f9d4c436, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682405720141 2023-04-25 06:55:23,318 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6111d21e29ac459bb1945cc9bdb178d5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405722188 2023-04-25 06:55:23,327 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:23,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:23,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:23,341 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#C#compaction#324 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:23,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:23,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:23,352 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:23,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:23,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:23,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,354 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:23,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:23,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:23,377 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,377 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:23,377 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:23,378 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:23,379 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405783370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405783370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,381 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,381 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:23,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23820338[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:23,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:23,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,392 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:23,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:23,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:23,392 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,392 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:23,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:23,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,395 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:23,395 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:23,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:23,399 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:23,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:23,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:23,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:23,399 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:23,400 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,400 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,400 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,401 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,403 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,405 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,406 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,406 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:23,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:23,406 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:23,406 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:23,507 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:23,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:23,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:23,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:23,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:23,510 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:23,510 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,510 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:23,510 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:23,511 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,511 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,513 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,513 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,513 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:23,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:23,514 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,514 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:23,515 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:23,515 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:23,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,516 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5662eb8c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:23,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:23,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:23,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,520 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:23,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:23,522 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:23,522 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:23,522 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:23,522 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:23,522 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:23,525 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,525 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,525 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,525 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,527 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,528 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,528 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,528 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,528 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,528 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,528 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:23,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:23,528 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,529 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:23,529 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:23,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405783581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405783581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,629 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:23,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:23,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:23,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:23,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:23,633 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:23,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:23,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:23,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,641 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,641 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:23,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:23,643 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,643 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:23,643 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:23,643 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:23,645 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4357ee50[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:23,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:23,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,648 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:23,650 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,650 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:23,652 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:23,654 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:23,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:23,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:23,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:23,654 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:23,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,657 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:23,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:23,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,663 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,663 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:23,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,663 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:23,675 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/edccbcc33bf04f0cac350deef9931e85 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/edccbcc33bf04f0cac350deef9931e85 2023-04-25 06:55:23,679 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/B of e275b2866a7bddedc0137edd1a42070e into edccbcc33bf04f0cac350deef9931e85(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:23,679 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:23,679 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/B, priority=13, startTime=1682405723205; duration=0sec 2023-04-25 06:55:23,680 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:23,698 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/ba329e7da90c484c921a90779caa683c 2023-04-25 06:55:23,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/82e1b748dfc44a6086693ce3a1b05af3 2023-04-25 06:55:23,763 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:23,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:23,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:23,766 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:23,766 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,767 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:23,767 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:23,767 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,767 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,768 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,768 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,768 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,769 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:23,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:23,769 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:23,770 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,770 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:23,770 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:23,770 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:23,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,771 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:23,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe38686[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:23,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:23,773 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,774 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:23,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:23,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:23,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,781 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/d1ce610454984adeb95fd66533c5a768 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d1ce610454984adeb95fd66533c5a768 2023-04-25 06:55:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,784 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/C of e275b2866a7bddedc0137edd1a42070e into d1ce610454984adeb95fd66533c5a768(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:23,784 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:23,784 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/C, priority=13, startTime=1682405723205; duration=0sec 2023-04-25 06:55:23,785 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:23,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:23,785 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:23,785 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:23,785 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:23,785 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:23,785 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:23,786 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,788 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:23,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:23,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,791 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:23,792 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:23,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405783886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:23,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405783886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,892 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:23,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:23,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:23,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:23,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:23,895 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:23,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:23,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:23,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,896 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,896 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,897 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,897 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:23,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:23,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:23,898 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,898 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:23,898 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:23,898 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,899 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:23,899 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:23,903 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c4bcb81[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:23,906 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:23,906 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:23,907 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,907 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:23,908 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,908 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:23,908 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,908 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:23,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:23,910 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:23,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:23,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:23,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:23,910 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:23,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,912 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:23,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,912 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:23,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:23,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:23,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:23,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:23,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:23,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:23,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:23,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:23,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:23,917 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,917 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:23,917 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:23,917 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:23,917 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:24,017 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:24,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:24,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:24,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:24,021 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:24,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:24,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:24,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,021 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,023 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,023 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:24,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:24,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:24,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:24,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:24,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:24,026 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db46d0f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:24,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:24,029 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,030 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:24,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:24,034 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:24,035 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:24,035 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:24,035 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:24,035 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:24,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,036 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,036 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,037 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:24,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:24,042 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:24,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,043 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:24,043 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:24,043 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,043 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:24,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/32c8add487784d719d8feb3b0abb2970 2023-04-25 06:55:24,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/ba329e7da90c484c921a90779caa683c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ba329e7da90c484c921a90779caa683c 2023-04-25 06:55:24,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ba329e7da90c484c921a90779caa683c, entries=150, sequenceid=264, filesize=25.4 K 2023-04-25 06:55:24,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/82e1b748dfc44a6086693ce3a1b05af3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/82e1b748dfc44a6086693ce3a1b05af3 2023-04-25 06:55:24,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/82e1b748dfc44a6086693ce3a1b05af3, entries=150, sequenceid=264, filesize=11.9 K 2023-04-25 06:55:24,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/32c8add487784d719d8feb3b0abb2970 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/32c8add487784d719d8feb3b0abb2970 2023-04-25 06:55:24,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/32c8add487784d719d8feb3b0abb2970, entries=150, sequenceid=264, filesize=11.9 K 2023-04-25 06:55:24,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e275b2866a7bddedc0137edd1a42070e in 906ms, sequenceid=264, compaction requested=false 2023-04-25 06:55:24,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:24,143 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:24,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:24,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:24,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:24,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:24,146 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:24,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:24,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:24,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,149 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,149 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,149 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,149 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:24,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:24,151 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:24,151 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:24,151 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:24,151 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:24,165 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d5d7b93c7b7e4389b10bcf32b2452e4a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d5d7b93c7b7e4389b10bcf32b2452e4a 2023-04-25 06:55:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:24,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405724328, max timestamp=1682405724347Num uniques 859; 2023-04-25 06:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405724328, max timestamp=1682405724347Num uniques 859; 2023-04-25 06:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405724328, max timestamp=1682405724347Num uniques 859; 2023-04-25 06:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:24,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405784382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405784392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405784394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405784414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405784414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405784515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405784521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405784522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,571 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/764f30728ab64fe8a07cbed05dde6004 2023-04-25 06:55:24,581 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/2f201327559844c0a92b3bce67b69dda 2023-04-25 06:55:24,589 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/4f71e63749b141cfa8d4c7fede80d729 2023-04-25 06:55:24,596 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/764f30728ab64fe8a07cbed05dde6004 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/764f30728ab64fe8a07cbed05dde6004 2023-04-25 06:55:24,608 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/764f30728ab64fe8a07cbed05dde6004, entries=150, sequenceid=277, filesize=25.4 K 2023-04-25 06:55:24,609 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/2f201327559844c0a92b3bce67b69dda as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/2f201327559844c0a92b3bce67b69dda 2023-04-25 06:55:24,614 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/2f201327559844c0a92b3bce67b69dda, entries=150, sequenceid=277, filesize=11.9 K 2023-04-25 06:55:24,614 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/4f71e63749b141cfa8d4c7fede80d729 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4f71e63749b141cfa8d4c7fede80d729 2023-04-25 06:55:24,622 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4f71e63749b141cfa8d4c7fede80d729, entries=150, sequenceid=277, filesize=11.9 K 2023-04-25 06:55:24,623 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for e275b2866a7bddedc0137edd1a42070e in 472ms, sequenceid=277, compaction requested=true 2023-04-25 06:55:24,623 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:24,623 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:24,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:24,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:24,623 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:24,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,624 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,625 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,625 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@216d2d87[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:24,627 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:24,627 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,628 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,629 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:24,629 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:24,629 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:24,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:24,630 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:24,630 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,630 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,630 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:24,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:24,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:24,632 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:24,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:24,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:24,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:24,632 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:24,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,636 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,636 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,638 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:24,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,640 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:24,640 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:24,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:24,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:24,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:24,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-25 06:55:24,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:24,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:24,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:24,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:24,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:24,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:24,740 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:24,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:24,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:24,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405784734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405784735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:24,744 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:24,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:24,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:24,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405784739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,745 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:24,748 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252cbc8a7170484a8b849ccba30ba6b4d4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252cbc8a7170484a8b849ccba30ba6b4d4 2023-04-25 06:55:24,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:24,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:24,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:24,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:24,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,758 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:24,767 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:24,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:24,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:24,772 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:24,772 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:24,774 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,775 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:24,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1209589a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:24,778 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:24,778 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,780 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:24,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:24,781 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:24,787 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:24,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:24,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:24,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:24,787 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:24,789 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,789 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,789 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,795 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,795 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,795 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:24,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:24,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:24,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,795 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:24,795 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:24,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:24,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405784845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405784845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405784845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,895 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:24,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:24,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:24,900 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:24,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:24,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:24,900 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,903 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:24,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:24,905 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,905 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:24,905 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:24,905 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:24,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dc8e99[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:24,908 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:24,909 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,909 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,910 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:24,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:24,910 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:24,910 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:24,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:24,913 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:24,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:24,913 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:24,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:24,914 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:24,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:24,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:24,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:24,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:24,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:24,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:24,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:24,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:24,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:24,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:24,920 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:24,920 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:24,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:24,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:24,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:24,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,020 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,024 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,025 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,025 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,026 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,027 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,027 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,027 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,027 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,027 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,038 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,038 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,038 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,038 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2efe5d98[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,040 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,040 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,041 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,041 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,042 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,044 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,045 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,046 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,046 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,047 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,047 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,052 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,053 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,053 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:25,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405785051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405785053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405785053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,153 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,156 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,156 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,156 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,156 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,157 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,157 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,158 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,159 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,159 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,159 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,160 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,160 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,160 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,160 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,161 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,161 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,162 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,162 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66374cd5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,163 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,164 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,164 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,165 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,165 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,168 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,170 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,170 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,170 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,170 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,170 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,172 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,172 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/9036edf968894e6fbca6840720628e8b 2023-04-25 06:55:25,172 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,172 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,173 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:25,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,176 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,177 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,177 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,177 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:25,176 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,177 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,277 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,280 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,281 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,281 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,282 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,283 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,283 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,283 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,283 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,286 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,286 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,288 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea21501[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,288 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,293 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,293 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,294 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,295 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:25,298 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,298 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,299 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,299 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:25,299 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405785359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405785359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405785359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,399 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,402 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,403 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,403 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,404 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,404 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,405 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,406 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,406 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,406 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,406 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,407 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a3eec0a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,412 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,416 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,419 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,419 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,419 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,419 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,419 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,421 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,427 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,427 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,428 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405785424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,428 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:25,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405785427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,528 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,532 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,532 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,533 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,533 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,534 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,534 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,536 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,536 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,536 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,536 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,539 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,539 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f07248[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,547 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,547 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,547 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,547 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,547 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,548 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,548 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,548 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,550 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,555 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:25,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,556 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,556 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:25,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/31335b3affd44009b68d334adb691db2 2023-04-25 06:55:25,656 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,660 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,661 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,661 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,663 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,663 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,663 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,664 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,665 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,665 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,666 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@afdcda7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,670 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,670 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,670 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,676 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,681 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,681 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,686 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,687 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:25,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:25,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,787 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,791 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,792 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,794 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,794 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,794 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2459dbe4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,801 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,803 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,805 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,805 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,811 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,814 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,814 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:25,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:25,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405785866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405785869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405785870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,914 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:25,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:25,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:25,917 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:25,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:25,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:25,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,919 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:25,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:25,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:25,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:25,921 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,921 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:25,921 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:25,921 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:25,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:25,922 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,923 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,923 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:25,924 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:25,924 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@597a2dd9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:25,926 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:25,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:25,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,927 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:25,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:25,930 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:25,930 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:25,930 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:25,930 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:25,930 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:25,931 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,931 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,931 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:25,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:25,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:25,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:25,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:25,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:25,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:25,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:25,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:25,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:25,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:25,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:25,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:25,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:25,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:25,939 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:25,939 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:26,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/65928d50d1d042df86cd150fb55e0611 2023-04-25 06:55:26,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/9036edf968894e6fbca6840720628e8b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/9036edf968894e6fbca6840720628e8b 2023-04-25 06:55:26,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/9036edf968894e6fbca6840720628e8b, entries=200, sequenceid=314, filesize=32.3 K 2023-04-25 06:55:26,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/31335b3affd44009b68d334adb691db2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/31335b3affd44009b68d334adb691db2 2023-04-25 06:55:26,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/31335b3affd44009b68d334adb691db2, entries=150, sequenceid=314, filesize=11.9 K 2023-04-25 06:55:26,022 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/65928d50d1d042df86cd150fb55e0611 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/65928d50d1d042df86cd150fb55e0611 2023-04-25 06:55:26,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/65928d50d1d042df86cd150fb55e0611, entries=150, sequenceid=314, filesize=11.9 K 2023-04-25 06:55:26,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for e275b2866a7bddedc0137edd1a42070e in 1302ms, sequenceid=314, compaction requested=true 2023-04-25 06:55:26,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:26,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:26,026 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:26,026 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:26,026 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:26,026 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:26,027 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111484 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:26,027 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:26,027 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/A is initiating minor compaction (all files) 2023-04-25 06:55:26,027 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/B is initiating minor compaction (all files) 2023-04-25 06:55:26,027 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/A in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:26,027 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/B in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:26,027 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/aaba3cbad9b3497fa4f6c5755e2367b4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ba329e7da90c484c921a90779caa683c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/764f30728ab64fe8a07cbed05dde6004, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/9036edf968894e6fbca6840720628e8b] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=108.9 K 2023-04-25 06:55:26,027 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/edccbcc33bf04f0cac350deef9931e85, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/82e1b748dfc44a6086693ce3a1b05af3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/2f201327559844c0a92b3bce67b69dda, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/31335b3affd44009b68d334adb691db2] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=47.9 K 2023-04-25 06:55:26,027 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting aaba3cbad9b3497fa4f6c5755e2367b4, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405720142 2023-04-25 06:55:26,028 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting edccbcc33bf04f0cac350deef9931e85, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405720142 2023-04-25 06:55:26,028 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting ba329e7da90c484c921a90779caa683c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682405722591 2023-04-25 06:55:26,028 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 82e1b748dfc44a6086693ce3a1b05af3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682405722591 2023-04-25 06:55:26,028 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 764f30728ab64fe8a07cbed05dde6004, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682405723253 2023-04-25 06:55:26,028 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2f201327559844c0a92b3bce67b69dda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682405723253 2023-04-25 06:55:26,028 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9036edf968894e6fbca6840720628e8b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405724379 2023-04-25 06:55:26,028 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 31335b3affd44009b68d334adb691db2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405724379 2023-04-25 06:55:26,039 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:26,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:26,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:26,043 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#B#compaction#334 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:26,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:26,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:26,045 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#A#compaction#333 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:26,045 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:26,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:26,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:26,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:26,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:26,045 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:26,047 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:26,047 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:26,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:26,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:26,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:26,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:26,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:26,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:26,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:26,055 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:26,055 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:26,055 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:26,060 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:26,060 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:26,060 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:26,060 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:26,060 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 06:55:26,061 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:26,061 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:26,061 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:26,061 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:26,061 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:26,061 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:26,083 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ea580421305640ca9fd7e70a3ed65e21 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ea580421305640ca9fd7e70a3ed65e21 2023-04-25 06:55:26,468 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/aab10713504d49ca86921edf02a26fe2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/aab10713504d49ca86921edf02a26fe2 2023-04-25 06:55:26,473 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/2ecbad86ca9b47e89b4c6826e012f6fb as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2ecbad86ca9b47e89b4c6826e012f6fb 2023-04-25 06:55:26,474 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/B of e275b2866a7bddedc0137edd1a42070e into aab10713504d49ca86921edf02a26fe2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:26,474 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:26,474 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/B, priority=12, startTime=1682405726026; duration=0sec 2023-04-25 06:55:26,475 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:26,475 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:26,477 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:26,477 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/C is initiating minor compaction (all files) 2023-04-25 06:55:26,477 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/C in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:26,477 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d1ce610454984adeb95fd66533c5a768, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/32c8add487784d719d8feb3b0abb2970, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4f71e63749b141cfa8d4c7fede80d729, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/65928d50d1d042df86cd150fb55e0611] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=47.9 K 2023-04-25 06:55:26,477 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting d1ce610454984adeb95fd66533c5a768, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405720142 2023-04-25 06:55:26,478 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 32c8add487784d719d8feb3b0abb2970, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682405722591 2023-04-25 06:55:26,478 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4f71e63749b141cfa8d4c7fede80d729, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682405723253 2023-04-25 06:55:26,478 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 65928d50d1d042df86cd150fb55e0611, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405724379 2023-04-25 06:55:26,482 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/A of e275b2866a7bddedc0137edd1a42070e into 2ecbad86ca9b47e89b4c6826e012f6fb(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:26,482 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:26,482 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/A, priority=12, startTime=1682405726025; duration=0sec 2023-04-25 06:55:26,483 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:26,490 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/53bc4915693840eab76092556fe7aba3 2023-04-25 06:55:26,507 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#C#compaction#336 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:26,558 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/48f4b7a640f34dae885985ed6aa5bfa6 2023-04-25 06:55:26,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:26,886 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:26,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405726911, max timestamp=1682405726919Num uniques 859; 2023-04-25 06:55:26,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405726911, max timestamp=1682405726919Num uniques 859; 2023-04-25 06:55:26,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405726911, max timestamp=1682405726919Num uniques 859; 2023-04-25 06:55:26,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:26,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:26,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:26,962 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/559c2e27c0a94b41919f66641b686964 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/559c2e27c0a94b41919f66641b686964 2023-04-25 06:55:26,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:26,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:26,966 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/C of e275b2866a7bddedc0137edd1a42070e into 559c2e27c0a94b41919f66641b686964(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:26,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405786959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:26,966 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:26,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405786958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:26,967 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/C, priority=12, startTime=1682405726026; duration=0sec 2023-04-25 06:55:26,967 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:26,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:26,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405786965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:26,994 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/acc9f5bc152640cab41606db2a49917a 2023-04-25 06:55:26,998 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/53bc4915693840eab76092556fe7aba3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53bc4915693840eab76092556fe7aba3 2023-04-25 06:55:27,011 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53bc4915693840eab76092556fe7aba3, entries=150, sequenceid=322, filesize=25.4 K 2023-04-25 06:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/48f4b7a640f34dae885985ed6aa5bfa6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/48f4b7a640f34dae885985ed6aa5bfa6 2023-04-25 06:55:27,016 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/48f4b7a640f34dae885985ed6aa5bfa6, entries=150, sequenceid=322, filesize=11.9 K 2023-04-25 06:55:27,017 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/acc9f5bc152640cab41606db2a49917a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/acc9f5bc152640cab41606db2a49917a 2023-04-25 06:55:27,023 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/acc9f5bc152640cab41606db2a49917a, entries=150, sequenceid=322, filesize=11.9 K 2023-04-25 06:55:27,024 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for e275b2866a7bddedc0137edd1a42070e in 964ms, sequenceid=322, compaction requested=false 2023-04-25 06:55:27,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:27,024 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,026 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ccd668c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,028 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,028 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,029 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,029 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,030 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,031 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,031 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,031 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,031 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,031 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,034 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,035 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,037 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,040 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,041 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,041 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,041 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,041 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-04-25 06:55:27,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:27,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 06:55:27,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:27,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 06:55:27,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:27,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 06:55:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:27,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405787075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405787078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405787079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,142 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:27,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:27,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:27,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:27,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:27,146 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:27,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:27,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:27,146 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,148 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,149 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:27,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:27,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,152 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,154 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b90247[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,154 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,156 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,157 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,157 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,158 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,161 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,161 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,161 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,163 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,163 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,163 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,164 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,165 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,171 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:27,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,171 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,171 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,171 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,172 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,172 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405787187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405787187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405787188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,272 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:27,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:27,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:27,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:27,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:27,275 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:27,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:27,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:27,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:27,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:27,279 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,280 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:27,280 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:27,280 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,281 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,282 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@255624f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,284 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,284 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,285 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,293 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,293 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,295 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,306 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,306 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405787389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405787389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405787395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,407 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:27,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:27,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:27,410 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:27,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:27,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:27,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,411 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:27,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:27,413 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,413 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:27,414 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:27,414 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,416 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a498339[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,418 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,418 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,419 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,419 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,419 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,423 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,423 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,425 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,425 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,425 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,430 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,430 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,431 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,431 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,431 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,431 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,431 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38688 deadline: 1682405787447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405787448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,454 DEBUG [Thread-2220] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:27,454 DEBUG [Thread-2224] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:27,510 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c678d8f8c46d4a0d8ec6feba37a7ee5b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c678d8f8c46d4a0d8ec6feba37a7ee5b 2023-04-25 06:55:27,532 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:27,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:27,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:27,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:27,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:27,535 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:27,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:27,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:27,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,538 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,538 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:27,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:27,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:27,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:27,540 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,540 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:27,540 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:27,540 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,542 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd76dfa[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,544 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,544 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,547 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,551 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,551 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,552 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,552 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,553 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,553 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,555 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,561 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,561 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,561 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:27,562 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,562 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,662 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:27,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:27,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:27,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:27,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:27,666 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:27,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:27,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:27,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:27,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:27,670 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,670 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:27,671 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:27,671 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5ce13d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,675 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,676 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,677 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,679 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:27,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,686 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,686 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,686 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405787695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405787696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:27,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405787701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,787 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:27,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:27,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:27,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:27,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:27,790 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:27,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:27,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:27,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,791 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:27,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6432a5c3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,802 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,803 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,806 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,806 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,807 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,807 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,807 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,812 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,812 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,812 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,812 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,812 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,812 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,812 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,812 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,912 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:27,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:27,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:27,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:27,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:27,915 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:27,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:27,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:27,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,918 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:27,918 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=363, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/50ea41ea0e644e0e9e59223d531e9f0c 2023-04-25 06:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:27,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:27,921 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,921 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:27,921 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:27,922 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:27,931 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a4a237[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:27,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:27,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:27,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:27,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:27,952 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:27,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:27,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:27,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:27,952 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:27,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,956 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:27,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,959 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,960 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:27,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:27,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:27,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:27,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:27,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:27,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:27,965 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:27,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:27,966 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:27,966 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:27,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:27,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:27,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:27,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:27,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/bbbe954584b34f498cea6fcff68520f2 2023-04-25 06:55:28,066 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:28,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:28,070 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:28,070 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,070 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:28,070 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:28,071 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,071 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,072 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:28,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:28,079 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744aa324[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:28,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:28,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,083 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:28,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:28,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:28,089 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:28,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:28,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:28,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:28,089 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:28,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:28,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:28,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,099 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:28,099 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:28,099 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,199 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:28,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:28,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:28,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:28,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:28,203 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:28,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:28,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:28,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,204 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,204 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:28,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38630 deadline: 1682405788201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:28,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:28,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405788201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:28,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:28,207 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,207 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:28,207 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:28,207 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:28,208 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:28,209 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,209 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e35f199[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:28,212 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:28,212 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405788208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,215 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:28,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,215 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:28,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:28,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:28,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:28,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:28,220 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:28,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:28,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:28,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:28,220 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:28,222 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,222 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,222 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:28,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,228 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:28,228 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:28,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:28,328 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:28,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:28,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:28,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:28,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:28,331 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:28,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:28,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:28,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,332 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,333 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,333 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,334 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,334 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:28,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:28,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:28,335 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,335 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:28,335 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:28,335 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:28,337 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,337 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,337 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f585b21[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:28,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:28,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:28,340 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:28,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:28,341 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:28,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:28,343 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:28,343 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:28,343 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:28,343 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:28,343 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:28,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,349 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,349 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:28,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,349 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:28,350 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:28,350 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:28,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/a7a3a68de838464f94af5afbee1c46d1 2023-04-25 06:55:28,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/50ea41ea0e644e0e9e59223d531e9f0c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/50ea41ea0e644e0e9e59223d531e9f0c 2023-04-25 06:55:28,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/50ea41ea0e644e0e9e59223d531e9f0c, entries=200, sequenceid=363, filesize=32.3 K 2023-04-25 06:55:28,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/bbbe954584b34f498cea6fcff68520f2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/bbbe954584b34f498cea6fcff68520f2 2023-04-25 06:55:28,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/bbbe954584b34f498cea6fcff68520f2, entries=150, sequenceid=363, filesize=11.9 K 2023-04-25 06:55:28,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/a7a3a68de838464f94af5afbee1c46d1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a7a3a68de838464f94af5afbee1c46d1 2023-04-25 06:55:28,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a7a3a68de838464f94af5afbee1c46d1, entries=150, sequenceid=363, filesize=11.9 K 2023-04-25 06:55:28,401 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=26.84 KB/27480 for e275b2866a7bddedc0137edd1a42070e in 1331ms, sequenceid=363, compaction requested=true 2023-04-25 06:55:28,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:28,401 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:28,401 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:28,402 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85724 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:28,402 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/A is initiating minor compaction (all files) 2023-04-25 06:55:28,402 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:28,402 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/A in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,403 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/B is initiating minor compaction (all files) 2023-04-25 06:55:28,403 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2ecbad86ca9b47e89b4c6826e012f6fb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53bc4915693840eab76092556fe7aba3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/50ea41ea0e644e0e9e59223d531e9f0c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=83.7 K 2023-04-25 06:55:28,403 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/B in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,403 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/aab10713504d49ca86921edf02a26fe2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/48f4b7a640f34dae885985ed6aa5bfa6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/bbbe954584b34f498cea6fcff68520f2] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=36.2 K 2023-04-25 06:55:28,403 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 2ecbad86ca9b47e89b4c6826e012f6fb, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405724379 2023-04-25 06:55:28,403 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting aab10713504d49ca86921edf02a26fe2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405724379 2023-04-25 06:55:28,403 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 53bc4915693840eab76092556fe7aba3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682405724732 2023-04-25 06:55:28,403 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 48f4b7a640f34dae885985ed6aa5bfa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682405724732 2023-04-25 06:55:28,404 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 50ea41ea0e644e0e9e59223d531e9f0c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682405726950 2023-04-25 06:55:28,404 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting bbbe954584b34f498cea6fcff68520f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682405726950 2023-04-25 06:55:28,411 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#B#compaction#343 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:28,414 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#A#compaction#342 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:28,450 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:28,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:28,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:28,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:28,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:28,455 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:28,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:28,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:28,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,456 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,457 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,457 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,457 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:28,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:28,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:28,459 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:28,459 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:28,470 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425a145e1b5a9884fa7838f730e48f97787 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a145e1b5a9884fa7838f730e48f97787 2023-04-25 06:55:28,475 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/12fbbe0bbb5b43759a7c8c904119fed9 2023-04-25 06:55:28,483 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/6ee2a6d2b5194da98bc2cc43f7c0eade 2023-04-25 06:55:28,517 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/8bc2076678854f318637eb2500bbc6ab 2023-04-25 06:55:28,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/12fbbe0bbb5b43759a7c8c904119fed9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/12fbbe0bbb5b43759a7c8c904119fed9 2023-04-25 06:55:28,524 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/12fbbe0bbb5b43759a7c8c904119fed9, entries=100, sequenceid=370, filesize=18.6 K 2023-04-25 06:55:28,525 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/6ee2a6d2b5194da98bc2cc43f7c0eade as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/6ee2a6d2b5194da98bc2cc43f7c0eade 2023-04-25 06:55:28,528 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/6ee2a6d2b5194da98bc2cc43f7c0eade, entries=100, sequenceid=370, filesize=9.5 K 2023-04-25 06:55:28,529 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/8bc2076678854f318637eb2500bbc6ab as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8bc2076678854f318637eb2500bbc6ab 2023-04-25 06:55:28,532 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8bc2076678854f318637eb2500bbc6ab, entries=100, sequenceid=370, filesize=9.5 K 2023-04-25 06:55:28,533 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e275b2866a7bddedc0137edd1a42070e in 74ms, sequenceid=370, compaction requested=true 2023-04-25 06:55:28,533 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:28,533 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28eaf1e7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:28,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:28,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,541 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,542 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,542 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:28,543 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:28,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:28,546 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:28,546 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:28,546 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:28,546 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:28,546 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:28,547 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,547 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,547 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,548 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,550 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,553 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,553 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:28,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,554 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,554 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,554 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,554 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,554 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:28,554 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,554 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:28,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,554 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,554 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,554 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,654 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:28,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:28,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:28,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:28,658 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:28,658 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,658 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:28,658 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:28,659 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,659 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:28,660 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:28,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:28,662 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,663 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:28,663 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:28,663 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:28,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:28,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@998245d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:28,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:28,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,675 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:28,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:28,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:28,675 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:28,677 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:28,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:28,680 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:28,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:28,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:28,680 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:28,680 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:28,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:28,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:28,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:28,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,687 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:28,687 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,787 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:28,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:28,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:28,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:28,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:28,792 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:28,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:28,792 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:28,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,793 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:28,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:28,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:28,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:28,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:28,796 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:28,799 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:28,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d0a5ce[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:28,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:28,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,802 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:28,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,803 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:28,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:28,806 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:28,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:28,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:28,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:28,806 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:28,807 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,807 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:28,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:28,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:28,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,813 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:28,813 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:28,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,813 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,823 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/69a5d5f1000547f5b19382c9e34f28bb as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/69a5d5f1000547f5b19382c9e34f28bb 2023-04-25 06:55:28,827 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/B of e275b2866a7bddedc0137edd1a42070e into 69a5d5f1000547f5b19382c9e34f28bb(size=12.5 K), total size for store is 22.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:28,828 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:28,828 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/B, priority=13, startTime=1682405728401; duration=0sec 2023-04-25 06:55:28,828 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:28,828 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:28,829 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:28,829 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/C is initiating minor compaction (all files) 2023-04-25 06:55:28,829 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/C in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,829 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/559c2e27c0a94b41919f66641b686964, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/acc9f5bc152640cab41606db2a49917a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a7a3a68de838464f94af5afbee1c46d1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8bc2076678854f318637eb2500bbc6ab] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=45.7 K 2023-04-25 06:55:28,830 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 559c2e27c0a94b41919f66641b686964, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682405724379 2023-04-25 06:55:28,830 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting acc9f5bc152640cab41606db2a49917a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682405724732 2023-04-25 06:55:28,831 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a7a3a68de838464f94af5afbee1c46d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682405726950 2023-04-25 06:55:28,831 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 8bc2076678854f318637eb2500bbc6ab, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682405727077 2023-04-25 06:55:28,837 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#C#compaction#347 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:28,857 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/4c788b12a8b6435182da6bd69ce7cfd9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4c788b12a8b6435182da6bd69ce7cfd9 2023-04-25 06:55:28,861 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/C of e275b2866a7bddedc0137edd1a42070e into 4c788b12a8b6435182da6bd69ce7cfd9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:28,861 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:28,861 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/C, priority=12, startTime=1682405728401; duration=0sec 2023-04-25 06:55:28,861 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:28,913 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:28,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:28,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:28,918 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:28,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:28,918 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:28,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:28,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:28,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:28,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:28,925 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:28,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:28,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:28,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:28,928 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,928 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15bc68a6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:28,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:28,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,932 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:28,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:28,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:28,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:28,962 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:28,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:28,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:28,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:28,962 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:28,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:28,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:28,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:28,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:28,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:28,970 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:28,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:28,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:28,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:28,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:28,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:28,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:28,971 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:28,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:28,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:28,971 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,071 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:29,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:29,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:29,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:29,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:29,075 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:29,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:29,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:29,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,077 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:29,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:29,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:29,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:29,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:29,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:29,079 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,079 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:29,080 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:29,080 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:29,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:29,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:29,080 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,081 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa6c63f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:29,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:29,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:29,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:29,084 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:29,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:29,086 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:29,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:29,088 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:29,088 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:29,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:29,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:29,089 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:29,091 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,091 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,091 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:29,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:29,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:29,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,098 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:29,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,097 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,098 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,198 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:29,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:29,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:29,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:29,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:29,209 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:29,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:29,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:29,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,209 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,211 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:29,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:29,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:29,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:29,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:29,214 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,215 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:29,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d4bdfa[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:29,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:29,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:29,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:29,219 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:29,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:29,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:29,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:29,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:29,223 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:29,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:29,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:29,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:29,223 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:29,225 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,225 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,225 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,234 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:29,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:29,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,235 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:29,235 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,235 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,326 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:55:29,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:29,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:29,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:29,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:29,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:29,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:29,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:29,335 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:29,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:29,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:29,340 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:29,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:29,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:29,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,352 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:29,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:29,360 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,360 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:29,360 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:29,360 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:29,362 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,363 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:29,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5782bb[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:29,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:29,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:29,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:29,366 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:29,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,368 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:29,369 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:29,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:29,373 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:29,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:29,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:29,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:29,373 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:29,376 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,376 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,377 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,378 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,385 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,385 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,385 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,385 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,386 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,386 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,386 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,386 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,386 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:29,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:29,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,386 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:29,387 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405729409, max timestamp=1682405729423Num uniques 859; 2023-04-25 06:55:29,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405729409, max timestamp=1682405729423Num uniques 859; 2023-04-25 06:55:29,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:29,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405729409, max timestamp=1682405729423Num uniques 859; 2023-04-25 06:55:29,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:29,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:29,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38630 deadline: 1682405789475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405789476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,487 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:29,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:29,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405789483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:29,490 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:29,490 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,490 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:29,490 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:29,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,491 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,492 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,492 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:29,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:29,493 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,493 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:29,493 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:29,493 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:29,496 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:29,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@295838af[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:29,500 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:29,500 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,501 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:29,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:29,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:29,501 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,502 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:29,505 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:29,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:29,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:29,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:29,506 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:29,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,508 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,509 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:29,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:29,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,513 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,514 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:29,514 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,514 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,514 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,514 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,514 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,514 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38630 deadline: 1682405789584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405789587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405789590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,614 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:29,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:29,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:29,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:29,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:29,617 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:29,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:29,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:29,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,617 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,618 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,618 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,619 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,619 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:29,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:29,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:29,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:29,620 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:29,622 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,622 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,622 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,622 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:29,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75939485[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:29,623 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:29,623 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,624 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:29,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:29,624 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:29,624 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,625 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:29,627 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:29,629 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:29,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:29,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:29,629 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:29,629 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:29,631 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,631 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,631 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,631 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,632 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,634 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:29,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,634 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:29,635 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:29,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,635 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,649 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/0251e62522c5457b99e65ebd204b7db0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/0251e62522c5457b99e65ebd204b7db0 2023-04-25 06:55:29,653 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/A of e275b2866a7bddedc0137edd1a42070e into 0251e62522c5457b99e65ebd204b7db0(size=26.1 K), total size for store is 44.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 06:55:29,653 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:29,653 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/A, priority=13, startTime=1682405728401; duration=1sec 2023-04-25 06:55:29,653 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:29,735 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:29,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:29,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:29,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:29,738 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,739 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,739 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,739 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,739 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:29,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:29,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:29,744 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:29,744 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,744 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:29,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:29,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:29,746 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:29,752 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,752 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,756 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d15e9fc3884b461bb1762b88a5d90c4e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d15e9fc3884b461bb1762b88a5d90c4e 2023-04-25 06:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72afcca0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:29,758 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:29,758 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,759 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:29,760 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,760 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:29,762 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:29,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:29,765 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:29,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:29,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:29,765 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:29,765 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:29,766 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,766 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,766 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,767 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,768 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,774 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:29,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,775 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,775 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:29,775 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38630 deadline: 1682405789787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38644 deadline: 1682405789797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:29,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405789799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,875 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:29,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:29,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:29,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:29,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:29,879 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:29,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:29,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:29,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,880 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,880 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,881 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,881 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:29,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:29,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:29,882 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:29,882 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:29,882 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:29,882 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:29,882 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:29,882 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,882 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:29,882 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:29,882 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,884 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@118eda14[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,886 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:29,885 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:29,886 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,887 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:29,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:29,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:29,887 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,887 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:29,889 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:29,892 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:29,892 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:29,892 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:29,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:29,892 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:29,892 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:29,894 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:29,894 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,894 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,896 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:29,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:29,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:29,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:29,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:29,898 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:29,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:29,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:29,899 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,899 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:29,899 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:29,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:29,899 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:29,899 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:29,999 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,002 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,003 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,003 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,003 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,003 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,003 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,004 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,004 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,006 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,006 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,006 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,006 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,008 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,009 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,009 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79fb05ca[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,010 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,010 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,011 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,013 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,016 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,016 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,016 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,016 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,016 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,017 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,017 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,018 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,018 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,019 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,022 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,022 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,023 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,023 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,023 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,023 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,023 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,023 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38630 deadline: 1682405790093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:30,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38644 deadline: 1682405790102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682405790110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,123 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,127 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,127 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,127 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,127 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,127 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,128 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,129 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,130 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,131 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,132 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,132 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,132 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,132 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,133 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,133 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@102591cc[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,135 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,136 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,136 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,139 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:30,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,142 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,142 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,142 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,143 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,144 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,144 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,145 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,147 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,147 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,147 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,147 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:30,147 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,148 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,148 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,148 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,163 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/692b708055424b51a0ea74994a30fa2e 2023-04-25 06:55:30,248 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,251 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,252 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,252 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,253 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,253 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,253 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,253 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,256 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,256 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,256 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,256 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,266 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,266 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce395a4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,269 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,270 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,273 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,273 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,279 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:30,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,280 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,280 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,380 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,384 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,385 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,386 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,386 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,389 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bdab256[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,392 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,394 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,394 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,394 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,394 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,394 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,397 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,397 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,397 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,398 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,401 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:30,402 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,402 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,502 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,505 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,506 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,506 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,508 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,508 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,508 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,509 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,511 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2e3db2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,512 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,512 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,513 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,515 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,515 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:30,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,518 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,518 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,521 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,529 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,529 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,529 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/35f4c82c1c0b4ec7af7bbcb0a5e81324 2023-04-25 06:55:30,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:30,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38630 deadline: 1682405790601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:30,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38644 deadline: 1682405790608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682405790616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,629 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,633 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,633 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,634 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,634 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,635 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,635 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,636 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,636 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,636 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,636 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,638 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,638 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b96cca[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,640 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,645 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,645 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,646 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,646 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,647 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,647 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:30,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,653 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,653 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,658 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,659 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,659 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,659 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,759 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,762 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,763 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,769 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,769 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,769 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,770 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,771 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,771 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,772 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,773 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,773 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f79b0c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,775 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,775 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,776 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,778 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,778 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,778 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,778 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:30,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,780 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,780 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,780 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,783 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,783 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,785 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,787 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,787 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,787 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,788 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,888 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:30,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:30,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:30,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:30,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:30,891 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:30,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:30,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:30,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,892 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,892 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,893 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,893 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:30,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:30,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:30,894 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,894 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:30,894 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:30,894 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:30,896 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:30,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4811dd86[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:30,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:30,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,902 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:30,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:30,903 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:30,905 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:30,906 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:30,906 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:30,906 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:30,906 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:30,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:30,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:30,908 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,909 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:30,911 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:30,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:30,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:30,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:30,911 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:30,911 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:30,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:30,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/b8858ca1c8c148438a7445184cd0bc8b 2023-04-25 06:55:30,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/692b708055424b51a0ea74994a30fa2e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/692b708055424b51a0ea74994a30fa2e 2023-04-25 06:55:30,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/692b708055424b51a0ea74994a30fa2e, entries=150, sequenceid=387, filesize=25.4 K 2023-04-25 06:55:30,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/35f4c82c1c0b4ec7af7bbcb0a5e81324 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/35f4c82c1c0b4ec7af7bbcb0a5e81324 2023-04-25 06:55:31,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/35f4c82c1c0b4ec7af7bbcb0a5e81324, entries=50, sequenceid=387, filesize=7.1 K 2023-04-25 06:55:31,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/b8858ca1c8c148438a7445184cd0bc8b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b8858ca1c8c148438a7445184cd0bc8b 2023-04-25 06:55:31,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b8858ca1c8c148438a7445184cd0bc8b, entries=50, sequenceid=387, filesize=7.1 K 2023-04-25 06:55:31,008 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e275b2866a7bddedc0137edd1a42070e in 1682ms, sequenceid=387, compaction requested=true 2023-04-25 06:55:31,008 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:31,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:31,009 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:31,009 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:31,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:31,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:31,010 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71732 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:31,010 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 29876 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:31,010 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/A is initiating minor compaction (all files) 2023-04-25 06:55:31,010 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e275b2866a7bddedc0137edd1a42070e/B is initiating minor compaction (all files) 2023-04-25 06:55:31,010 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/A in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:31,010 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e275b2866a7bddedc0137edd1a42070e/B in TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:31,010 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/0251e62522c5457b99e65ebd204b7db0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/12fbbe0bbb5b43759a7c8c904119fed9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/692b708055424b51a0ea74994a30fa2e] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=70.1 K 2023-04-25 06:55:31,010 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/69a5d5f1000547f5b19382c9e34f28bb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/6ee2a6d2b5194da98bc2cc43f7c0eade, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/35f4c82c1c0b4ec7af7bbcb0a5e81324] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp, totalSize=29.2 K 2023-04-25 06:55:31,010 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 0251e62522c5457b99e65ebd204b7db0, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682405726950 2023-04-25 06:55:31,010 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 69a5d5f1000547f5b19382c9e34f28bb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682405726950 2023-04-25 06:55:31,011 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 12fbbe0bbb5b43759a7c8c904119fed9, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682405727077 2023-04-25 06:55:31,011 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6ee2a6d2b5194da98bc2cc43f7c0eade, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682405727077 2023-04-25 06:55:31,011 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 692b708055424b51a0ea74994a30fa2e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682405729299 2023-04-25 06:55:31,011 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 35f4c82c1c0b4ec7af7bbcb0a5e81324, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682405729324 2023-04-25 06:55:31,012 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:31,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:31,023 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#B#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:31,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:31,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:31,024 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:31,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:31,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:31,024 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:31,025 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:31,025 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:31,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,032 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:31,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:31,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:31,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:31,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:31,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:31,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:31,036 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:31,036 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:31,037 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:31,037 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:31,037 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:31,037 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:31,037 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. started... 2023-04-25 06:55:31,037 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-25 06:55:31,037 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:31,037 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:31,038 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:31,038 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:31,038 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:31,038 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:31,058 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e275b2866a7bddedc0137edd1a42070e#A#compaction#352 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:31,463 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/a1b089e3841c4135a1daccbd31a40e09 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/a1b089e3841c4135a1daccbd31a40e09 2023-04-25 06:55:31,468 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/B of e275b2866a7bddedc0137edd1a42070e into a1b089e3841c4135a1daccbd31a40e09(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:31,468 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:31,468 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/B, priority=13, startTime=1682405731009; duration=0sec 2023-04-25 06:55:31,468 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:31,468 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-25 06:55:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:31,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. as already flushing 2023-04-25 06:55:31,469 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-25 06:55:31,469 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-25 06:55:31,469 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. because compaction request was cancelled 2023-04-25 06:55:31,483 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042537ccf0294cca45939fee969ab0fff280 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042537ccf0294cca45939fee969ab0fff280 2023-04-25 06:55:31,503 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/cd3c66e30e40483e871ec22a59bedeca as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/cd3c66e30e40483e871ec22a59bedeca 2023-04-25 06:55:31,508 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e275b2866a7bddedc0137edd1a42070e/A of e275b2866a7bddedc0137edd1a42070e into cd3c66e30e40483e871ec22a59bedeca(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:31,508 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:31,508 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e., storeName=e275b2866a7bddedc0137edd1a42070e/A, priority=13, startTime=1682405731008; duration=0sec 2023-04-25 06:55:31,508 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:31,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38688 deadline: 1682405791509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405791510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38630 deadline: 1682405791609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38688 deadline: 1682405791619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38644 deadline: 1682405791619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405791619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682405791624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,708 DEBUG [Thread-2235] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72eaaf6e to 127.0.0.1:60416 2023-04-25 06:55:31,708 DEBUG [Thread-2235] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:31,711 DEBUG [Thread-2237] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4130667f to 127.0.0.1:60416 2023-04-25 06:55:31,711 DEBUG [Thread-2237] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:31,713 DEBUG [Thread-2239] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b73a361 to 127.0.0.1:60416 2023-04-25 06:55:31,713 DEBUG [Thread-2239] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:31,716 DEBUG [Thread-2231] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x678b4c3d to 127.0.0.1:60416 2023-04-25 06:55:31,716 DEBUG [Thread-2231] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:31,719 DEBUG [Thread-2233] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43736f95 to 127.0.0.1:60416 2023-04-25 06:55:31,719 DEBUG [Thread-2233] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:31,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38688 deadline: 1682405791826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405791826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:31,900 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=423, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/ef35b2d95fc94f2187ede46965a69cbb 2023-04-25 06:55:32,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:32,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38688 deadline: 1682405792128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:32,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405792128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,309 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/86602b7f1dbe4ca791989193c2719c95 2023-04-25 06:55:32,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:32,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38688 deadline: 1682405792629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:32,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38690 deadline: 1682405792633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,718 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/5f7ee73255014c75b33949ab423e7058 2023-04-25 06:55:32,721 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/ef35b2d95fc94f2187ede46965a69cbb as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ef35b2d95fc94f2187ede46965a69cbb 2023-04-25 06:55:32,723 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ef35b2d95fc94f2187ede46965a69cbb, entries=150, sequenceid=423, filesize=25.4 K 2023-04-25 06:55:32,724 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/86602b7f1dbe4ca791989193c2719c95 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/86602b7f1dbe4ca791989193c2719c95 2023-04-25 06:55:32,727 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/86602b7f1dbe4ca791989193c2719c95, entries=150, sequenceid=423, filesize=11.9 K 2023-04-25 06:55:32,728 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/5f7ee73255014c75b33949ab423e7058 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/5f7ee73255014c75b33949ab423e7058 2023-04-25 06:55:32,731 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/5f7ee73255014c75b33949ab423e7058, entries=150, sequenceid=423, filesize=11.9 K 2023-04-25 06:55:32,731 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for e275b2866a7bddedc0137edd1a42070e in 1694ms, sequenceid=423, compaction requested=true 2023-04-25 06:55:32,731 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:32,732 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:32,732 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:32,732 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:32,732 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,732 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:32,732 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:32,733 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,733 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:32,734 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:32,734 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:32,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:32,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f9aa9b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:32,735 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:32,735 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,739 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:32,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:32,740 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:32,740 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,740 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:32,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,741 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:32,741 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:32,741 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:32,742 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:32,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:32,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:32,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:32,742 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:32,744 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:32,744 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:32,744 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:32,744 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:32,745 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:32,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:32,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:32,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:32,747 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:32,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,747 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:32,747 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:32,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:32,747 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:32,847 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:32,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:33,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:33,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 06:55:33,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:33,618 DEBUG [Thread-2228] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63a535f3 to 127.0.0.1:60416 2023-04-25 06:55:33,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:33,618 DEBUG [Thread-2228] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:33,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:33,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:33,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:33,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:33,630 DEBUG [Thread-2222] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2329054b to 127.0.0.1:60416 2023-04-25 06:55:33,630 DEBUG [Thread-2222] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:33,637 DEBUG [Thread-2220] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d800352 to 127.0.0.1:60416 2023-04-25 06:55:33,637 DEBUG [Thread-2220] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:33,637 DEBUG [Thread-2226] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5baa6768 to 127.0.0.1:60416 2023-04-25 06:55:33,637 DEBUG [Thread-2226] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:33,638 DEBUG [Thread-2224] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30951255 to 127.0.0.1:60416 2023-04-25 06:55:33,638 DEBUG [Thread-2224] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1706 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5113 rows 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1710 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5130 rows 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1706 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5118 rows 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1695 2023-04-25 06:55:33,639 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5084 rows 2023-04-25 06:55:33,640 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1702 2023-04-25 06:55:33,640 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5105 rows 2023-04-25 06:55:33,640 INFO [Listener at localhost/41663] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 06:55:33,640 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x494529f9 to 127.0.0.1:60416 2023-04-25 06:55:33,640 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:55:33,641 INFO [Listener at localhost/41663] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 06:55:33,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 06:55:33,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:33,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 06:55:33,646 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405733646"}]},"ts":"1682405733646"} 2023-04-25 06:55:33,647 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 06:55:33,648 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 06:55:33,649 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, UNASSIGN}] 2023-04-25 06:55:33,650 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, UNASSIGN 2023-04-25 06:55:33,651 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:33,651 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405733651"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405733651"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405733651"}]},"ts":"1682405733651"} 2023-04-25 06:55:33,652 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:55:33,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 06:55:33,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:33,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e275b2866a7bddedc0137edd1a42070e, disabling compactions & flushes 2023-04-25 06:55:33,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:33,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 06:55:34,029 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425756852176abf4f3692eeb92cfea7424b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425756852176abf4f3692eeb92cfea7424b 2023-04-25 06:55:34,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 06:55:34,434 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=436, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/227849f580344c8683333082399886fd 2023-04-25 06:55:34,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 06:55:34,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/94f6a3cdb652404b8deab7ec739ed20c 2023-04-25 06:55:34,873 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/f0ee3d5e1a364146b857fbca10f261cf 2023-04-25 06:55:34,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/227849f580344c8683333082399886fd as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/227849f580344c8683333082399886fd 2023-04-25 06:55:34,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/227849f580344c8683333082399886fd, entries=150, sequenceid=436, filesize=25.4 K 2023-04-25 06:55:34,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/94f6a3cdb652404b8deab7ec739ed20c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/94f6a3cdb652404b8deab7ec739ed20c 2023-04-25 06:55:34,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/94f6a3cdb652404b8deab7ec739ed20c, entries=150, sequenceid=436, filesize=11.9 K 2023-04-25 06:55:34,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/f0ee3d5e1a364146b857fbca10f261cf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f0ee3d5e1a364146b857fbca10f261cf 2023-04-25 06:55:34,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f0ee3d5e1a364146b857fbca10f261cf, entries=150, sequenceid=436, filesize=11.9 K 2023-04-25 06:55:34,891 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for e275b2866a7bddedc0137edd1a42070e in 1273ms, sequenceid=436, compaction requested=true 2023-04-25 06:55:34,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:34,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:34,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:34,891 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. because compaction request was cancelled 2023-04-25 06:55:34,891 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. because compaction request was cancelled 2023-04-25 06:55:34,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:34,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:34,891 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. because compaction request was cancelled 2023-04-25 06:55:34,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. after waiting 0 ms 2023-04-25 06:55:34,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:34,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:34,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e275b2866a7bddedc0137edd1a42070e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 06:55:34,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=A 2023-04-25 06:55:34,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:34,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=B 2023-04-25 06:55:34,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:34,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e275b2866a7bddedc0137edd1a42070e, store=C 2023-04-25 06:55:34,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:35,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425eba83c9ca5a346b596e4c29bcecb84fe to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425eba83c9ca5a346b596e4c29bcecb84fe 2023-04-25 06:55:35,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=443, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/6642754c0f4b449dbc9f23874e60a416 2023-04-25 06:55:35,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 06:55:36,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/08c47d101eb04ff496588e78831359b8 2023-04-25 06:55:36,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/b49851ae4a9a48058208afffd349d1be 2023-04-25 06:55:36,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/A/6642754c0f4b449dbc9f23874e60a416 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/6642754c0f4b449dbc9f23874e60a416 2023-04-25 06:55:36,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/6642754c0f4b449dbc9f23874e60a416, entries=150, sequenceid=443, filesize=25.4 K 2023-04-25 06:55:36,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/B/08c47d101eb04ff496588e78831359b8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/08c47d101eb04ff496588e78831359b8 2023-04-25 06:55:36,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/08c47d101eb04ff496588e78831359b8, entries=150, sequenceid=443, filesize=11.9 K 2023-04-25 06:55:36,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/.tmp/C/b49851ae4a9a48058208afffd349d1be as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b49851ae4a9a48058208afffd349d1be 2023-04-25 06:55:36,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b49851ae4a9a48058208afffd349d1be, entries=150, sequenceid=443, filesize=11.9 K 2023-04-25 06:55:36,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e275b2866a7bddedc0137edd1a42070e in 1650ms, sequenceid=443, compaction requested=true 2023-04-25 06:55:36,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/def1891eda3c49f79fae19274b57ee7c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53e6a8872674490f8b3d04a327d5ea15, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c7d515ee6f2541b99dbbe54320ca5f29, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/bd63fe0e25754b7aa9fde16945403bda, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63857662ec68439daa2127bd11a0a75c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/b3e4895e5390443a899f0ed105b90634, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c16c9e8619674915af62b393459ebff9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/5f7440b0ae0a4409a103b83d2e96b555, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/21bcec1ae4a44a178b7f33cd4067ef40, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/57b9e83b702e4ac3b2f5d169b6c85a8f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2f7d3513be7f41458c537f5eeb65f0a3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/aaba3cbad9b3497fa4f6c5755e2367b4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63cde6bab0e141da8f1392d940a8ef9d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ba329e7da90c484c921a90779caa683c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/764f30728ab64fe8a07cbed05dde6004, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/9036edf968894e6fbca6840720628e8b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2ecbad86ca9b47e89b4c6826e012f6fb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53bc4915693840eab76092556fe7aba3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/50ea41ea0e644e0e9e59223d531e9f0c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/0251e62522c5457b99e65ebd204b7db0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/12fbbe0bbb5b43759a7c8c904119fed9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/692b708055424b51a0ea74994a30fa2e] to archive 2023-04-25 06:55:36,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:36,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/def1891eda3c49f79fae19274b57ee7c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/def1891eda3c49f79fae19274b57ee7c 2023-04-25 06:55:36,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53e6a8872674490f8b3d04a327d5ea15 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53e6a8872674490f8b3d04a327d5ea15 2023-04-25 06:55:36,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c7d515ee6f2541b99dbbe54320ca5f29 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c7d515ee6f2541b99dbbe54320ca5f29 2023-04-25 06:55:36,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/bd63fe0e25754b7aa9fde16945403bda to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/bd63fe0e25754b7aa9fde16945403bda 2023-04-25 06:55:36,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63857662ec68439daa2127bd11a0a75c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63857662ec68439daa2127bd11a0a75c 2023-04-25 06:55:36,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/b3e4895e5390443a899f0ed105b90634 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/b3e4895e5390443a899f0ed105b90634 2023-04-25 06:55:36,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c16c9e8619674915af62b393459ebff9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/c16c9e8619674915af62b393459ebff9 2023-04-25 06:55:36,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/5f7440b0ae0a4409a103b83d2e96b555 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/5f7440b0ae0a4409a103b83d2e96b555 2023-04-25 06:55:36,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/21bcec1ae4a44a178b7f33cd4067ef40 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/21bcec1ae4a44a178b7f33cd4067ef40 2023-04-25 06:55:36,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/57b9e83b702e4ac3b2f5d169b6c85a8f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/57b9e83b702e4ac3b2f5d169b6c85a8f 2023-04-25 06:55:36,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2f7d3513be7f41458c537f5eeb65f0a3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2f7d3513be7f41458c537f5eeb65f0a3 2023-04-25 06:55:36,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/aaba3cbad9b3497fa4f6c5755e2367b4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/aaba3cbad9b3497fa4f6c5755e2367b4 2023-04-25 06:55:36,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63cde6bab0e141da8f1392d940a8ef9d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/63cde6bab0e141da8f1392d940a8ef9d 2023-04-25 06:55:36,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ba329e7da90c484c921a90779caa683c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ba329e7da90c484c921a90779caa683c 2023-04-25 06:55:36,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/764f30728ab64fe8a07cbed05dde6004 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/764f30728ab64fe8a07cbed05dde6004 2023-04-25 06:55:36,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/9036edf968894e6fbca6840720628e8b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/9036edf968894e6fbca6840720628e8b 2023-04-25 06:55:36,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2ecbad86ca9b47e89b4c6826e012f6fb to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/2ecbad86ca9b47e89b4c6826e012f6fb 2023-04-25 06:55:36,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53bc4915693840eab76092556fe7aba3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/53bc4915693840eab76092556fe7aba3 2023-04-25 06:55:36,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/50ea41ea0e644e0e9e59223d531e9f0c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/50ea41ea0e644e0e9e59223d531e9f0c 2023-04-25 06:55:36,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/0251e62522c5457b99e65ebd204b7db0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/0251e62522c5457b99e65ebd204b7db0 2023-04-25 06:55:36,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/12fbbe0bbb5b43759a7c8c904119fed9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/12fbbe0bbb5b43759a7c8c904119fed9 2023-04-25 06:55:36,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/692b708055424b51a0ea74994a30fa2e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/692b708055424b51a0ea74994a30fa2e 2023-04-25 06:55:36,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/0014d1a4d16440638f66e025057bb2f2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/d63471f3a654453fb0c0e827151e7133, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/fcad0a138b304a62a32096a16f3a10fd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/7bd8d2779c2d45219c4fcc4f177e537a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/3037885800814008ade22f55d1cfe374, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/429aa264a2b24f218fd42853bb6be71f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/992d8ce99cc5460392e957b0611b51a0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/b734aaa5ba7d48998e6162f6b796e6a0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/33ca28ca442e4830a7fcc3d0f16e6e45, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/19655ab151774980815b78431acafcad, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/c1013d3e67724a24b94267febd45be0f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/edccbcc33bf04f0cac350deef9931e85, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/e2de162ad4a542e4901b380eae9c0ebc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/82e1b748dfc44a6086693ce3a1b05af3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/2f201327559844c0a92b3bce67b69dda, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/aab10713504d49ca86921edf02a26fe2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/31335b3affd44009b68d334adb691db2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/48f4b7a640f34dae885985ed6aa5bfa6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/69a5d5f1000547f5b19382c9e34f28bb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/bbbe954584b34f498cea6fcff68520f2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/6ee2a6d2b5194da98bc2cc43f7c0eade, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/35f4c82c1c0b4ec7af7bbcb0a5e81324] to archive 2023-04-25 06:55:36,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:36,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/0014d1a4d16440638f66e025057bb2f2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/0014d1a4d16440638f66e025057bb2f2 2023-04-25 06:55:36,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/d63471f3a654453fb0c0e827151e7133 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/d63471f3a654453fb0c0e827151e7133 2023-04-25 06:55:36,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/fcad0a138b304a62a32096a16f3a10fd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/fcad0a138b304a62a32096a16f3a10fd 2023-04-25 06:55:36,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/7bd8d2779c2d45219c4fcc4f177e537a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/7bd8d2779c2d45219c4fcc4f177e537a 2023-04-25 06:55:36,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/3037885800814008ade22f55d1cfe374 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/3037885800814008ade22f55d1cfe374 2023-04-25 06:55:36,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/429aa264a2b24f218fd42853bb6be71f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/429aa264a2b24f218fd42853bb6be71f 2023-04-25 06:55:36,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/992d8ce99cc5460392e957b0611b51a0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/992d8ce99cc5460392e957b0611b51a0 2023-04-25 06:55:36,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/b734aaa5ba7d48998e6162f6b796e6a0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/b734aaa5ba7d48998e6162f6b796e6a0 2023-04-25 06:55:36,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/33ca28ca442e4830a7fcc3d0f16e6e45 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/33ca28ca442e4830a7fcc3d0f16e6e45 2023-04-25 06:55:36,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/19655ab151774980815b78431acafcad to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/19655ab151774980815b78431acafcad 2023-04-25 06:55:36,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/c1013d3e67724a24b94267febd45be0f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/c1013d3e67724a24b94267febd45be0f 2023-04-25 06:55:36,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/edccbcc33bf04f0cac350deef9931e85 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/edccbcc33bf04f0cac350deef9931e85 2023-04-25 06:55:36,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/e2de162ad4a542e4901b380eae9c0ebc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/e2de162ad4a542e4901b380eae9c0ebc 2023-04-25 06:55:36,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/82e1b748dfc44a6086693ce3a1b05af3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/82e1b748dfc44a6086693ce3a1b05af3 2023-04-25 06:55:36,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/2f201327559844c0a92b3bce67b69dda to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/2f201327559844c0a92b3bce67b69dda 2023-04-25 06:55:36,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/aab10713504d49ca86921edf02a26fe2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/aab10713504d49ca86921edf02a26fe2 2023-04-25 06:55:36,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/31335b3affd44009b68d334adb691db2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/31335b3affd44009b68d334adb691db2 2023-04-25 06:55:36,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/48f4b7a640f34dae885985ed6aa5bfa6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/48f4b7a640f34dae885985ed6aa5bfa6 2023-04-25 06:55:36,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/69a5d5f1000547f5b19382c9e34f28bb to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/69a5d5f1000547f5b19382c9e34f28bb 2023-04-25 06:55:36,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/bbbe954584b34f498cea6fcff68520f2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/bbbe954584b34f498cea6fcff68520f2 2023-04-25 06:55:36,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/6ee2a6d2b5194da98bc2cc43f7c0eade to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/6ee2a6d2b5194da98bc2cc43f7c0eade 2023-04-25 06:55:36,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/35f4c82c1c0b4ec7af7bbcb0a5e81324 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/35f4c82c1c0b4ec7af7bbcb0a5e81324 2023-04-25 06:55:36,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/37ae7af6c0294b1fb11ab3178be45a9b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/856a4c23294a494db14dd7931a9bf6f2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a590c13e4f8744458b6f8708aa31895c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/2aada0af044549f7a227edab4152793d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8395b33ff423440fbdf68c29cc963460, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4b8ba5f9331c4266ab4715cefcaf345b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f465cff9d9274e67ad505a488a1bd1f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/511d748cc2d14ed3bb3d0832e6339f40, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d2e48f04891a4345898fafc191dd0758, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/c5ef0798df014eb99e76c1ccecf6d1db, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/87b272cb4ebd4b858658bb50f9d4c436, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d1ce610454984adeb95fd66533c5a768, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/6111d21e29ac459bb1945cc9bdb178d5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/32c8add487784d719d8feb3b0abb2970, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4f71e63749b141cfa8d4c7fede80d729, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/559c2e27c0a94b41919f66641b686964, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/65928d50d1d042df86cd150fb55e0611, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/acc9f5bc152640cab41606db2a49917a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a7a3a68de838464f94af5afbee1c46d1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8bc2076678854f318637eb2500bbc6ab] to archive 2023-04-25 06:55:36,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:36,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/37ae7af6c0294b1fb11ab3178be45a9b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/37ae7af6c0294b1fb11ab3178be45a9b 2023-04-25 06:55:36,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/856a4c23294a494db14dd7931a9bf6f2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/856a4c23294a494db14dd7931a9bf6f2 2023-04-25 06:55:36,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a590c13e4f8744458b6f8708aa31895c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a590c13e4f8744458b6f8708aa31895c 2023-04-25 06:55:36,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/2aada0af044549f7a227edab4152793d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/2aada0af044549f7a227edab4152793d 2023-04-25 06:55:36,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8395b33ff423440fbdf68c29cc963460 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8395b33ff423440fbdf68c29cc963460 2023-04-25 06:55:36,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4b8ba5f9331c4266ab4715cefcaf345b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4b8ba5f9331c4266ab4715cefcaf345b 2023-04-25 06:55:36,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f465cff9d9274e67ad505a488a1bd1f7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f465cff9d9274e67ad505a488a1bd1f7 2023-04-25 06:55:36,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/511d748cc2d14ed3bb3d0832e6339f40 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/511d748cc2d14ed3bb3d0832e6339f40 2023-04-25 06:55:36,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d2e48f04891a4345898fafc191dd0758 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d2e48f04891a4345898fafc191dd0758 2023-04-25 06:55:36,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/c5ef0798df014eb99e76c1ccecf6d1db to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/c5ef0798df014eb99e76c1ccecf6d1db 2023-04-25 06:55:36,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/87b272cb4ebd4b858658bb50f9d4c436 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/87b272cb4ebd4b858658bb50f9d4c436 2023-04-25 06:55:36,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d1ce610454984adeb95fd66533c5a768 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/d1ce610454984adeb95fd66533c5a768 2023-04-25 06:55:36,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/6111d21e29ac459bb1945cc9bdb178d5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/6111d21e29ac459bb1945cc9bdb178d5 2023-04-25 06:55:36,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/32c8add487784d719d8feb3b0abb2970 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/32c8add487784d719d8feb3b0abb2970 2023-04-25 06:55:36,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4f71e63749b141cfa8d4c7fede80d729 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4f71e63749b141cfa8d4c7fede80d729 2023-04-25 06:55:36,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/559c2e27c0a94b41919f66641b686964 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/559c2e27c0a94b41919f66641b686964 2023-04-25 06:55:36,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/65928d50d1d042df86cd150fb55e0611 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/65928d50d1d042df86cd150fb55e0611 2023-04-25 06:55:36,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/acc9f5bc152640cab41606db2a49917a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/acc9f5bc152640cab41606db2a49917a 2023-04-25 06:55:36,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a7a3a68de838464f94af5afbee1c46d1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/a7a3a68de838464f94af5afbee1c46d1 2023-04-25 06:55:36,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8bc2076678854f318637eb2500bbc6ab to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/8bc2076678854f318637eb2500bbc6ab 2023-04-25 06:55:36,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/recovered.edits/446.seqid, newMaxSeqId=446, maxSeqId=4 2023-04-25 06:55:36,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e. 2023-04-25 06:55:36,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e275b2866a7bddedc0137edd1a42070e: 2023-04-25 06:55:36,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:36,627 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=e275b2866a7bddedc0137edd1a42070e, regionState=CLOSED 2023-04-25 06:55:36,628 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405736627"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405736627"}]},"ts":"1682405736627"} 2023-04-25 06:55:36,629 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-25 06:55:36,630 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure e275b2866a7bddedc0137edd1a42070e, server=jenkins-hbase4.apache.org,41021,1682405622759 in 2.9760 sec 2023-04-25 06:55:36,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-25 06:55:36,631 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e275b2866a7bddedc0137edd1a42070e, UNASSIGN in 2.9810 sec 2023-04-25 06:55:36,631 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405736631"}]},"ts":"1682405736631"} 2023-04-25 06:55:36,632 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 06:55:36,633 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 06:55:36,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9920 sec 2023-04-25 06:55:37,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 06:55:37,752 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-25 06:55:37,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 06:55:37,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:37,755 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:37,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-25 06:55:37,755 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:37,760 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:37,761 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/recovered.edits] 2023-04-25 06:55:37,764 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/227849f580344c8683333082399886fd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/227849f580344c8683333082399886fd 2023-04-25 06:55:37,765 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/6642754c0f4b449dbc9f23874e60a416 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/6642754c0f4b449dbc9f23874e60a416 2023-04-25 06:55:37,766 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/cd3c66e30e40483e871ec22a59bedeca to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/cd3c66e30e40483e871ec22a59bedeca 2023-04-25 06:55:37,767 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ef35b2d95fc94f2187ede46965a69cbb to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/A/ef35b2d95fc94f2187ede46965a69cbb 2023-04-25 06:55:37,769 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/08c47d101eb04ff496588e78831359b8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/08c47d101eb04ff496588e78831359b8 2023-04-25 06:55:37,770 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/86602b7f1dbe4ca791989193c2719c95 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/86602b7f1dbe4ca791989193c2719c95 2023-04-25 06:55:37,771 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/94f6a3cdb652404b8deab7ec739ed20c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/94f6a3cdb652404b8deab7ec739ed20c 2023-04-25 06:55:37,772 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/a1b089e3841c4135a1daccbd31a40e09 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/B/a1b089e3841c4135a1daccbd31a40e09 2023-04-25 06:55:37,774 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4c788b12a8b6435182da6bd69ce7cfd9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/4c788b12a8b6435182da6bd69ce7cfd9 2023-04-25 06:55:37,775 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/5f7ee73255014c75b33949ab423e7058 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/5f7ee73255014c75b33949ab423e7058 2023-04-25 06:55:37,776 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b49851ae4a9a48058208afffd349d1be to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b49851ae4a9a48058208afffd349d1be 2023-04-25 06:55:37,777 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b8858ca1c8c148438a7445184cd0bc8b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/b8858ca1c8c148438a7445184cd0bc8b 2023-04-25 06:55:37,779 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f0ee3d5e1a364146b857fbca10f261cf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/C/f0ee3d5e1a364146b857fbca10f261cf 2023-04-25 06:55:37,782 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/recovered.edits/446.seqid to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e/recovered.edits/446.seqid 2023-04-25 06:55:37,782 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e275b2866a7bddedc0137edd1a42070e 2023-04-25 06:55:37,782 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:55:37,783 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 06:55:37,783 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 06:55:37,786 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251c4f466e584b43848cbba12e7f58a54a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251c4f466e584b43848cbba12e7f58a54a 2023-04-25 06:55:37,788 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252b3859de0a1b4b6cad934ad6e19ca1b6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252b3859de0a1b4b6cad934ad6e19ca1b6 2023-04-25 06:55:37,789 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252cbc8a7170484a8b849ccba30ba6b4d4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252cbc8a7170484a8b849ccba30ba6b4d4 2023-04-25 06:55:37,790 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042537ccf0294cca45939fee969ab0fff280 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042537ccf0294cca45939fee969ab0fff280 2023-04-25 06:55:37,791 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425402aa40c0bf544279e4ad7d032b9e2a8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425402aa40c0bf544279e4ad7d032b9e2a8 2023-04-25 06:55:37,792 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042555f7ec3ac7a842f29f1923a55313a063 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042555f7ec3ac7a842f29f1923a55313a063 2023-04-25 06:55:37,793 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042564cb499521f249dfb3c3d3694b12c88f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042564cb499521f249dfb3c3d3694b12c88f 2023-04-25 06:55:37,794 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425756852176abf4f3692eeb92cfea7424b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425756852176abf4f3692eeb92cfea7424b 2023-04-25 06:55:37,795 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425910e49e697994e0799244499dfc157b3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425910e49e697994e0799244499dfc157b3 2023-04-25 06:55:37,797 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a145e1b5a9884fa7838f730e48f97787 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a145e1b5a9884fa7838f730e48f97787 2023-04-25 06:55:37,798 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c50bf27733f64e6fbf06205ed28e0dfd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c50bf27733f64e6fbf06205ed28e0dfd 2023-04-25 06:55:37,799 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c678d8f8c46d4a0d8ec6feba37a7ee5b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c678d8f8c46d4a0d8ec6feba37a7ee5b 2023-04-25 06:55:37,801 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d15e9fc3884b461bb1762b88a5d90c4e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d15e9fc3884b461bb1762b88a5d90c4e 2023-04-25 06:55:37,802 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d5d7b93c7b7e4389b10bcf32b2452e4a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d5d7b93c7b7e4389b10bcf32b2452e4a 2023-04-25 06:55:37,803 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d6bb5073e71548ef864a26da90c4551a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d6bb5073e71548ef864a26da90c4551a 2023-04-25 06:55:37,804 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ea580421305640ca9fd7e70a3ed65e21 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ea580421305640ca9fd7e70a3ed65e21 2023-04-25 06:55:37,805 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425eba83c9ca5a346b596e4c29bcecb84fe to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425eba83c9ca5a346b596e4c29bcecb84fe 2023-04-25 06:55:37,806 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ee514062ddb34e9c80a8624de6617169 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ee514062ddb34e9c80a8624de6617169 2023-04-25 06:55:37,807 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f0cedf9283d447a1899197d761d69422 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f0cedf9283d447a1899197d761d69422 2023-04-25 06:55:37,808 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425fbc9bd12bb914600b8713c834930ad7c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425fbc9bd12bb914600b8713c834930ad7c 2023-04-25 06:55:37,809 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 06:55:37,811 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:37,813 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 06:55:37,814 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 06:55:37,815 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:37,815 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 06:55:37,815 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682405737815"}]},"ts":"9223372036854775807"} 2023-04-25 06:55:37,817 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 06:55:37,817 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e275b2866a7bddedc0137edd1a42070e, NAME => 'TestAcidGuarantees,,1682405708638.e275b2866a7bddedc0137edd1a42070e.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 06:55:37,817 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 06:55:37,817 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682405737817"}]},"ts":"9223372036854775807"} 2023-04-25 06:55:37,818 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 06:55:37,820 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:37,821 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 68 msec 2023-04-25 06:55:37,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-25 06:55:37,856 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-25 06:55:37,868 INFO [Listener at localhost/41663] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 227), OpenFileDescriptor=549 (was 547) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 387), ProcessCount=172 (was 172), AvailableMemoryMB=4202 (was 4257) 2023-04-25 06:55:37,880 INFO [Listener at localhost/41663] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=549, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=172, AvailableMemoryMB=4202 2023-04-25 06:55:37,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:55:37,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:55:37,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 06:55:37,884 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 06:55:37,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-25 06:55:37,885 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 06:55:37,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 06:55:37,886 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:37,887 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4 empty. 2023-04-25 06:55:37,887 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:37,887 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:55:37,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 06:55:38,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 06:55:38,295 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:55:38,296 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 75fc43298851a5c4f1060516f9ac39d4, NAME => 'TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp 2023-04-25 06:55:38,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 06:55:38,712 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:55:38,712 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 75fc43298851a5c4f1060516f9ac39d4, disabling compactions & flushes 2023-04-25 06:55:38,712 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:38,712 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:38,712 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. after waiting 0 ms 2023-04-25 06:55:38,713 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:38,713 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:38,713 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:38,714 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 06:55:38,715 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405738715"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405738715"}]},"ts":"1682405738715"} 2023-04-25 06:55:38,716 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 06:55:38,717 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 06:55:38,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405738717"}]},"ts":"1682405738717"} 2023-04-25 06:55:38,717 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 06:55:38,721 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75fc43298851a5c4f1060516f9ac39d4, ASSIGN}] 2023-04-25 06:55:38,722 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75fc43298851a5c4f1060516f9ac39d4, ASSIGN 2023-04-25 06:55:38,723 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=75fc43298851a5c4f1060516f9ac39d4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=false 2023-04-25 06:55:38,874 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=75fc43298851a5c4f1060516f9ac39d4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:38,874 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405738874"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405738874"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405738874"}]},"ts":"1682405738874"} 2023-04-25 06:55:38,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:55:38,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 06:55:39,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:39,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 75fc43298851a5c4f1060516f9ac39d4, NAME => 'TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:55:39,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:55:39,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,031 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,033 DEBUG [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A 2023-04-25 06:55:39,033 DEBUG [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A 2023-04-25 06:55:39,033 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:39,033 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75fc43298851a5c4f1060516f9ac39d4 columnFamilyName A 2023-04-25 06:55:39,034 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.HStore(310): Store=75fc43298851a5c4f1060516f9ac39d4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:39,034 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,035 DEBUG [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B 2023-04-25 06:55:39,035 DEBUG [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B 2023-04-25 06:55:39,035 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:39,035 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75fc43298851a5c4f1060516f9ac39d4 columnFamilyName B 2023-04-25 06:55:39,036 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.HStore(310): Store=75fc43298851a5c4f1060516f9ac39d4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:39,036 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,037 DEBUG [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C 2023-04-25 06:55:39,037 DEBUG [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C 2023-04-25 06:55:39,037 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:55:39,037 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 75fc43298851a5c4f1060516f9ac39d4 columnFamilyName C 2023-04-25 06:55:39,038 INFO [StoreOpener-75fc43298851a5c4f1060516f9ac39d4-1] regionserver.HStore(310): Store=75fc43298851a5c4f1060516f9ac39d4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:55:39,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:39,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:55:39,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:39,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:55:39,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 75fc43298851a5c4f1060516f9ac39d4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61294366, jitterRate=-0.08664277195930481}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:55:39,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:39,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., pid=49, masterSystemTime=1682405739027 2023-04-25 06:55:39,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:39,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:39,045 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=75fc43298851a5c4f1060516f9ac39d4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:39,045 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405739045"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405739045"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405739045"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405739045"}]},"ts":"1682405739045"} 2023-04-25 06:55:39,047 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-25 06:55:39,047 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 in 171 msec 2023-04-25 06:55:39,049 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-25 06:55:39,049 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=75fc43298851a5c4f1060516f9ac39d4, ASSIGN in 327 msec 2023-04-25 06:55:39,049 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 06:55:39,049 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405739049"}]},"ts":"1682405739049"} 2023-04-25 06:55:39,050 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 06:55:39,052 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 06:55:39,052 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-25 06:55:39,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 06:55:39,990 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-25 06:55:39,991 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x2c8fbc35 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:39,995 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66369677, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:39,996 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:39,999 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56480, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:39,999 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 06:55:40,001 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36738, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 06:55:40,002 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x3fb41c50 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,006 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e0b6187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,006 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x357cb237 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,010 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ab56e54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,011 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4539ed0c to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,014 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15840296, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,014 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x32cd672d to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,018 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3816bdf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,018 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x09b8ff30 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,025 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50150525, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,025 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x028a3a77 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,030 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4e3b2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,030 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4cefb012 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,034 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f65627a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,034 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x24a917df to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,038 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15f74c78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,038 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x78fa3dee to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,042 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1401cac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,042 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x36d2d826 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:55:40,051 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f6e865f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:55:40,056 DEBUG [hconnection-0x2e8e7be7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,056 DEBUG [hconnection-0x7fa518b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,056 DEBUG [hconnection-0x6016122c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,058 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,058 DEBUG [hconnection-0x64c44334-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,058 DEBUG [hconnection-0x66f9e929-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,058 DEBUG [hconnection-0x715f62e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,058 DEBUG [hconnection-0x6680783a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:40,059 DEBUG [hconnection-0x554790d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,059 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56524, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,061 DEBUG [hconnection-0x5d0cb38b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,064 DEBUG [hconnection-0x6156ab96-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:55:40,064 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56488, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,064 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,064 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,065 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56538, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,066 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,067 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,069 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:40,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:40,069 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:40,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:40,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:40,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:40,071 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:40,073 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56546, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:55:40,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:40,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:40,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:40,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:40,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:40,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:40,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:40,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:40,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:40,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:40,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:40,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:40,079 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:40,079 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:40,080 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 06:55:40,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:40,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:40,081 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:40,081 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:40,081 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:40,081 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:40,081 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:40,081 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:40,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405800096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405740090, max timestamp=1682405740096Num uniques 859; 2023-04-25 06:55:40,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:40,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405740090, max timestamp=1682405740096Num uniques 859; 2023-04-25 06:55:40,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405740090, max timestamp=1682405740096Num uniques 859; 2023-04-25 06:55:40,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:40,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:55:40,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405800116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405800118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405800118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405800119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405800201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405800220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405800223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405800223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405800225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405800404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405800424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405800427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405800428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405800428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,493 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/bd10bc7ae32c48b88f5605a46172db05 2023-04-25 06:55:40,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405800710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405800728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405800731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405800731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:40,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405800733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,908 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/a6ac49107b84403c9c29493ade069a0d 2023-04-25 06:55:40,918 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/d5585fcc291b4a5b8ff72c12aeb3970b 2023-04-25 06:55:40,922 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/bd10bc7ae32c48b88f5605a46172db05 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bd10bc7ae32c48b88f5605a46172db05 2023-04-25 06:55:40,925 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bd10bc7ae32c48b88f5605a46172db05, entries=100, sequenceid=12, filesize=9.3 K 2023-04-25 06:55:40,926 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/a6ac49107b84403c9c29493ade069a0d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a6ac49107b84403c9c29493ade069a0d 2023-04-25 06:55:40,929 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a6ac49107b84403c9c29493ade069a0d, entries=100, sequenceid=12, filesize=9.3 K 2023-04-25 06:55:40,930 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/d5585fcc291b4a5b8ff72c12aeb3970b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d5585fcc291b4a5b8ff72c12aeb3970b 2023-04-25 06:55:40,933 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d5585fcc291b4a5b8ff72c12aeb3970b, entries=100, sequenceid=12, filesize=9.3 K 2023-04-25 06:55:40,934 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 75fc43298851a5c4f1060516f9ac39d4 in 854ms, sequenceid=12, compaction requested=false 2023-04-25 06:55:40,934 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 06:55:40,934 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:40,935 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:40,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:40,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:40,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:40,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:40,936 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:40,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:40,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e8b5512[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:40,938 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:40,938 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:40,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:40,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:40,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,939 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:40,941 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:40,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:40,944 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:40,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:40,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:40,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:40,944 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:40,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:40,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:40,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:40,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:40,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:40,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:40,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:40,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:40,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:40,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:40,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:40,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:40,950 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:40,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:40,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:40,950 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:40,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:40,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:40,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:40,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:41,050 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:41,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:41,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:41,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:41,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:41,054 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:41,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:41,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:41,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:41,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:41,054 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:41,055 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,055 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:41,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:41,055 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,055 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:41,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:41,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:41,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:41,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:41,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:41,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:41,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:41,057 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:41,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:41,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:41,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:41,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405801238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405801241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405801242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405801243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405801244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405801351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405801351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405801351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405801351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,466 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b52cdcbcfef74c578772ba32a50956be 2023-04-25 06:55:41,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405801559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405801559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405801560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405801560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405801863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405801867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405801868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:41,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405801870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:41,877 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/02b14c0120934e128fbbac2f83bb5951 2023-04-25 06:55:42,161 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 06:55:42,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405802247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,285 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/c181d63238b74968bcfdde2d3e915a0d 2023-04-25 06:55:42,290 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b52cdcbcfef74c578772ba32a50956be as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b52cdcbcfef74c578772ba32a50956be 2023-04-25 06:55:42,294 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b52cdcbcfef74c578772ba32a50956be, entries=150, sequenceid=47, filesize=11.6 K 2023-04-25 06:55:42,295 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/02b14c0120934e128fbbac2f83bb5951 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/02b14c0120934e128fbbac2f83bb5951 2023-04-25 06:55:42,298 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/02b14c0120934e128fbbac2f83bb5951, entries=150, sequenceid=47, filesize=11.6 K 2023-04-25 06:55:42,299 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/c181d63238b74968bcfdde2d3e915a0d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c181d63238b74968bcfdde2d3e915a0d 2023-04-25 06:55:42,302 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c181d63238b74968bcfdde2d3e915a0d, entries=150, sequenceid=47, filesize=11.6 K 2023-04-25 06:55:42,303 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 75fc43298851a5c4f1060516f9ac39d4 in 1246ms, sequenceid=47, compaction requested=false 2023-04-25 06:55:42,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:42,303 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:42,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:42,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:42,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:42,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,305 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,305 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:42,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@147e8834[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:42,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:42,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,308 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:42,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:42,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:42,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:42,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:42,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:42,316 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:42,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:42,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:42,316 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:42,316 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:42,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,318 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,319 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,322 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,322 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,322 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:42,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:42,322 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,322 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,322 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:42,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,323 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:42,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,323 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:42,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:42,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:55:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:42,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/0f86713745fe4d5bae9d47216028e12f 2023-04-25 06:55:42,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/855ee8dd550742d99626145943429ebc 2023-04-25 06:55:42,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405802407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405802408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405802409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405802410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,423 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:42,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:42,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:42,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:42,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:42,426 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:42,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:42,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:42,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,428 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:42,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:42,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:42,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:42,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:42,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:42,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:42,432 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,432 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:42,432 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:42,432 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:42,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:42,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:42,432 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:42,440 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,440 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,442 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,443 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:42,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:42,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52c58ad0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:42,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:42,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:42,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:42,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:42,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,446 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:42,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:42,450 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:42,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:42,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:42,450 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:42,450 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:42,451 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,451 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,452 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,453 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,455 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,458 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,459 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:42,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,459 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,459 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:42,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405802516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405802516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405802519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405802519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,559 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:42,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:42,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:42,562 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:42,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:42,563 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:42,563 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,563 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,564 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:42,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:42,566 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,566 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:42,566 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:42,566 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:42,567 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,569 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,569 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:42,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c464f61[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:42,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:42,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,572 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:42,572 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,572 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:42,574 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:42,578 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:42,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:42,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:42,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:42,578 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:42,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,581 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:42,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,585 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:42,585 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:42,585 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,586 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,686 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:42,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:42,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:42,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:42,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:42,689 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:42,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:42,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:42,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:42,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:42,692 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:42,692 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,692 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:42,693 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,693 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:42,693 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:42,693 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:42,696 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,697 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,698 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,698 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6871b63e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:42,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:42,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,700 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:42,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:42,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,702 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,702 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:42,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:42,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:42,703 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:42,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:42,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:42,703 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:42,703 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:42,707 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,707 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,707 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,709 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,711 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:42,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:42,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,712 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:42,712 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:42,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:42,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405802721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405802721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405802722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:42,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405802722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,812 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:42,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:42,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:42,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:42,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:42,818 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:42,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:42,818 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:42,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,820 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:42,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:42,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/01cf5c174c3345ccb9d509958aefe4e5 2023-04-25 06:55:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:42,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:42,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:42,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:42,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:42,826 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@82ebc2c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:42,830 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:42,830 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/0f86713745fe4d5bae9d47216028e12f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/0f86713745fe4d5bae9d47216028e12f 2023-04-25 06:55:42,832 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,832 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:42,832 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:42,832 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:42,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,832 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:42,833 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:42,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,834 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:42,834 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/0f86713745fe4d5bae9d47216028e12f, entries=200, sequenceid=59, filesize=13.9 K 2023-04-25 06:55:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/855ee8dd550742d99626145943429ebc as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/855ee8dd550742d99626145943429ebc 2023-04-25 06:55:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:42,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:42,837 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:42,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:42,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:42,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:42,837 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:42,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:42,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:42,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:42,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,845 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:42,845 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:42,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:42,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:42,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:42,845 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:42,845 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/855ee8dd550742d99626145943429ebc, entries=150, sequenceid=59, filesize=11.6 K 2023-04-25 06:55:42,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/01cf5c174c3345ccb9d509958aefe4e5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/01cf5c174c3345ccb9d509958aefe4e5 2023-04-25 06:55:42,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/01cf5c174c3345ccb9d509958aefe4e5, entries=150, sequenceid=59, filesize=11.6 K 2023-04-25 06:55:42,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 75fc43298851a5c4f1060516f9ac39d4 in 483ms, sequenceid=59, compaction requested=true 2023-04-25 06:55:42,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:42,859 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:42,859 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:55:42,860 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35654 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:42,860 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:55:42,861 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,861 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bd10bc7ae32c48b88f5605a46172db05, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b52cdcbcfef74c578772ba32a50956be, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/0f86713745fe4d5bae9d47216028e12f] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=34.8 K 2023-04-25 06:55:42,861 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:42,861 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:55:42,861 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,861 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a6ac49107b84403c9c29493ade069a0d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/02b14c0120934e128fbbac2f83bb5951, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/855ee8dd550742d99626145943429ebc] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=32.5 K 2023-04-25 06:55:42,861 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting bd10bc7ae32c48b88f5605a46172db05, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682405740073 2023-04-25 06:55:42,863 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a6ac49107b84403c9c29493ade069a0d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682405740073 2023-04-25 06:55:42,863 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b52cdcbcfef74c578772ba32a50956be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682405740114 2023-04-25 06:55:42,864 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 02b14c0120934e128fbbac2f83bb5951, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682405740114 2023-04-25 06:55:42,864 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 0f86713745fe4d5bae9d47216028e12f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682405741215 2023-04-25 06:55:42,865 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 855ee8dd550742d99626145943429ebc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682405741242 2023-04-25 06:55:42,875 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:42,875 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:42,899 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/ee49a5d4b3b5463388cdbc6c5aa86ecc as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ee49a5d4b3b5463388cdbc6c5aa86ecc 2023-04-25 06:55:42,905 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into ee49a5d4b3b5463388cdbc6c5aa86ecc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:42,905 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:42,905 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=13, startTime=1682405742859; duration=0sec 2023-04-25 06:55:42,905 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:42,905 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:42,907 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:42,907 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:55:42,907 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,907 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d5585fcc291b4a5b8ff72c12aeb3970b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c181d63238b74968bcfdde2d3e915a0d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/01cf5c174c3345ccb9d509958aefe4e5] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=32.5 K 2023-04-25 06:55:42,908 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d5585fcc291b4a5b8ff72c12aeb3970b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682405740073 2023-04-25 06:55:42,908 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c181d63238b74968bcfdde2d3e915a0d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682405740114 2023-04-25 06:55:42,908 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 01cf5c174c3345ccb9d509958aefe4e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682405741242 2023-04-25 06:55:42,929 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#373 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:42,945 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:42,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:42,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:42,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:42,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:42,961 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:42,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:42,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:42,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,963 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:42,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:42,964 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:42,964 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:42,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:42,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:42,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:42,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:42,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:42,968 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:42,968 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:42,968 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:42,976 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:42,976 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:42,976 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:55:42,976 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:42,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:42,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:42,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:42,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:42,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:43,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:43,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:43,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405803055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405803063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405803063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405803064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405803164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405803174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405803174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405803175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,293 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/cfc945f3e1d949ea9f7fd7fbddafaf96 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cfc945f3e1d949ea9f7fd7fbddafaf96 2023-04-25 06:55:43,297 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into cfc945f3e1d949ea9f7fd7fbddafaf96(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:43,297 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:43,297 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=13, startTime=1682405742859; duration=0sec 2023-04-25 06:55:43,297 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:43,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405803371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,375 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/c3ad46da3d7041c8ae97824615627cba as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c3ad46da3d7041c8ae97824615627cba 2023-04-25 06:55:43,379 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into c3ad46da3d7041c8ae97824615627cba(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:43,379 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:43,379 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=13, startTime=1682405742859; duration=0sec 2023-04-25 06:55:43,379 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:43,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405803386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405803386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405803388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,405 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/44f94e5eaef44081afdec9e25e075d9b 2023-04-25 06:55:43,429 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/4b09c574f203497a9b7ad554f72108b5 2023-04-25 06:55:43,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405803672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405803694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405803695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:43,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405803697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,851 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/36c8f90d8c79403685e91f7c3cc522ed 2023-04-25 06:55:43,858 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/44f94e5eaef44081afdec9e25e075d9b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/44f94e5eaef44081afdec9e25e075d9b 2023-04-25 06:55:43,863 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/44f94e5eaef44081afdec9e25e075d9b, entries=150, sequenceid=80, filesize=11.6 K 2023-04-25 06:55:43,864 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/4b09c574f203497a9b7ad554f72108b5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4b09c574f203497a9b7ad554f72108b5 2023-04-25 06:55:43,873 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4b09c574f203497a9b7ad554f72108b5, entries=150, sequenceid=80, filesize=11.6 K 2023-04-25 06:55:43,874 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/36c8f90d8c79403685e91f7c3cc522ed as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c8f90d8c79403685e91f7c3cc522ed 2023-04-25 06:55:43,886 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c8f90d8c79403685e91f7c3cc522ed, entries=150, sequenceid=80, filesize=11.6 K 2023-04-25 06:55:43,887 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 75fc43298851a5c4f1060516f9ac39d4 in 911ms, sequenceid=80, compaction requested=false 2023-04-25 06:55:43,887 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:43,887 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:43,889 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,889 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:43,889 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@764334f3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:43,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:43,891 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,893 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,893 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,893 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:43,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:43,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:43,894 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:43,895 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:43,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:43,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:43,898 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:43,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:43,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:43,898 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:43,898 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:43,901 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:43,901 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:43,901 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:43,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:43,903 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,905 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,905 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:43,905 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:43,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:43,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:43,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:43,905 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:43,905 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:43,905 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:43,905 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:43,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:43,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:43,905 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:43,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:43,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:43,906 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,006 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:44,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:44,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:44,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:44,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:44,009 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:44,009 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:44,009 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:44,009 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:44,010 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,010 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:44,011 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:44,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:44,011 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:44,012 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,012 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:44,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:44,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:44,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:44,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:44,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:44,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:44,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:44,013 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:44,013 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:44,013 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:44,013 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:55:44,013 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:44,013 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:44,024 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/4bb5d726da8e4c649dcc0edc65630ee1 2023-04-25 06:55:44,189 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bd10bc7ae32c48b88f5605a46172db05, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b52cdcbcfef74c578772ba32a50956be, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/0f86713745fe4d5bae9d47216028e12f] to archive 2023-04-25 06:55:44,190 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:44,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:44,191 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bd10bc7ae32c48b88f5605a46172db05 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bd10bc7ae32c48b88f5605a46172db05 2023-04-25 06:55:44,192 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b52cdcbcfef74c578772ba32a50956be to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b52cdcbcfef74c578772ba32a50956be 2023-04-25 06:55:44,194 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/0f86713745fe4d5bae9d47216028e12f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/0f86713745fe4d5bae9d47216028e12f 2023-04-25 06:55:44,200 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a6ac49107b84403c9c29493ade069a0d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/02b14c0120934e128fbbac2f83bb5951, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/855ee8dd550742d99626145943429ebc] to archive 2023-04-25 06:55:44,201 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:44,203 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a6ac49107b84403c9c29493ade069a0d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a6ac49107b84403c9c29493ade069a0d 2023-04-25 06:55:44,204 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/02b14c0120934e128fbbac2f83bb5951 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/02b14c0120934e128fbbac2f83bb5951 2023-04-25 06:55:44,206 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/855ee8dd550742d99626145943429ebc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/855ee8dd550742d99626145943429ebc 2023-04-25 06:55:44,213 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d5585fcc291b4a5b8ff72c12aeb3970b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c181d63238b74968bcfdde2d3e915a0d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/01cf5c174c3345ccb9d509958aefe4e5] to archive 2023-04-25 06:55:44,214 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:55:44,215 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d5585fcc291b4a5b8ff72c12aeb3970b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d5585fcc291b4a5b8ff72c12aeb3970b 2023-04-25 06:55:44,216 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c181d63238b74968bcfdde2d3e915a0d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c181d63238b74968bcfdde2d3e915a0d 2023-04-25 06:55:44,217 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/01cf5c174c3345ccb9d509958aefe4e5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/01cf5c174c3345ccb9d509958aefe4e5 2023-04-25 06:55:44,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405804226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405804227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405804228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405804230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405804268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,274 DEBUG [Thread-2940] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:44,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405804335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405804335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405804335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405804336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,435 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/ff560ff0543142a4ab1f3783071692ca 2023-04-25 06:55:44,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405804541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405804542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405804542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405804542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405804844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405804844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405804845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:44,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405804850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,866 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/92a81ce74b8a4cff991f243d5ad25b17 2023-04-25 06:55:44,870 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/4bb5d726da8e4c649dcc0edc65630ee1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bb5d726da8e4c649dcc0edc65630ee1 2023-04-25 06:55:44,874 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bb5d726da8e4c649dcc0edc65630ee1, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 06:55:44,881 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/ff560ff0543142a4ab1f3783071692ca as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ff560ff0543142a4ab1f3783071692ca 2023-04-25 06:55:44,884 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ff560ff0543142a4ab1f3783071692ca, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 06:55:44,885 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/92a81ce74b8a4cff991f243d5ad25b17 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/92a81ce74b8a4cff991f243d5ad25b17 2023-04-25 06:55:44,889 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/92a81ce74b8a4cff991f243d5ad25b17, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 06:55:44,889 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 75fc43298851a5c4f1060516f9ac39d4 in 876ms, sequenceid=95, compaction requested=true 2023-04-25 06:55:44,889 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:44,889 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:44,890 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,890 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:44,890 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:44,891 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b0401d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:44,893 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:44,893 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,895 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:44,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:44,895 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:44,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,895 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:44,896 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:44,899 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:44,899 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:44,899 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:44,899 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:44,899 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:44,900 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,900 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:44,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,900 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:44,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:44,901 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:44,902 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,910 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:44,910 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:44,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:44,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:44,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:44,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:44,910 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:44,910 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:44,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:44,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:44,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:44,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:44,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:44,911 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:44,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:45,010 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:45,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:45,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:45,014 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:45,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:45,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:45,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:45,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:45,014 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:45,015 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:45,015 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:45,016 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,016 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:45,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:45,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:45,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:45,017 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:45,017 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:45,017 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:55:45,018 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:45,018 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:45,018 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:45,018 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:45,018 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:45,018 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:45,368 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:45,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405805391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405805399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405805400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405805401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,436 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/bed1d94a8730432fa2d22dec66c40c27 2023-04-25 06:55:45,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405805502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405805509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405805509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405805512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405805713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405805716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405805717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405805723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:45,875 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/10d57afc62bd4745ac45d31b9ec67f06 2023-04-25 06:55:46,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405806017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405806022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405806023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405806030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,286 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/7ad2169899ee486cb7749b8caf32bf74 2023-04-25 06:55:46,290 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/bed1d94a8730432fa2d22dec66c40c27 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bed1d94a8730432fa2d22dec66c40c27 2023-04-25 06:55:46,293 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bed1d94a8730432fa2d22dec66c40c27, entries=150, sequenceid=114, filesize=11.6 K 2023-04-25 06:55:46,294 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/10d57afc62bd4745ac45d31b9ec67f06 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/10d57afc62bd4745ac45d31b9ec67f06 2023-04-25 06:55:46,297 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/10d57afc62bd4745ac45d31b9ec67f06, entries=150, sequenceid=114, filesize=11.6 K 2023-04-25 06:55:46,298 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/7ad2169899ee486cb7749b8caf32bf74 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7ad2169899ee486cb7749b8caf32bf74 2023-04-25 06:55:46,301 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7ad2169899ee486cb7749b8caf32bf74, entries=150, sequenceid=114, filesize=11.6 K 2023-04-25 06:55:46,302 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 75fc43298851a5c4f1060516f9ac39d4 in 1285ms, sequenceid=114, compaction requested=true 2023-04-25 06:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:46,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:46,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d15d173[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:46,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:46,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:46,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:46,309 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:46,310 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:46,312 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:46,312 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:46,312 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:46,312 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:46,312 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:46,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:46,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:46,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:46,314 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:46,315 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:46,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:46,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:46,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:46,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,318 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:46,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:46,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:46,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:46,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:46,318 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:46,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:46,318 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:46,318 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,318 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:46,418 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:46,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:46,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:46,423 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:46,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:46,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:46,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:46,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,427 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:46,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,428 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:46,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:46,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:46,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:46,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:46,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:46,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:46,430 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:46,430 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:46,431 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:55:46,431 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:46,431 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:46,431 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:46,431 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:46,431 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:46,431 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:46,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:46,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:46,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405806561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405806566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405806567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405806571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405806672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405806677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405806677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405806684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,842 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/1b589d13d996451a80c8cdf969c9179b 2023-04-25 06:55:46,852 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/cabec64139514e8481635a8ba9fdab18 2023-04-25 06:55:46,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405806879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405806881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405806881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:46,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:46,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405806888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405807186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405807187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405807187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405807195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,270 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/82a01267182549cd8bd4ed5c0795317b 2023-04-25 06:55:47,274 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/1b589d13d996451a80c8cdf969c9179b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1b589d13d996451a80c8cdf969c9179b 2023-04-25 06:55:47,277 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1b589d13d996451a80c8cdf969c9179b, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 06:55:47,278 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/cabec64139514e8481635a8ba9fdab18 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cabec64139514e8481635a8ba9fdab18 2023-04-25 06:55:47,282 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cabec64139514e8481635a8ba9fdab18, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 06:55:47,283 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/82a01267182549cd8bd4ed5c0795317b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/82a01267182549cd8bd4ed5c0795317b 2023-04-25 06:55:47,286 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/82a01267182549cd8bd4ed5c0795317b, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 06:55:47,287 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 75fc43298851a5c4f1060516f9ac39d4 in 856ms, sequenceid=127, compaction requested=true 2023-04-25 06:55:47,287 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:47,287 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:47,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:47,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:47,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2027d921[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:47,292 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:47,292 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,293 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:47,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:47,295 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:47,297 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:47,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:47,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:47,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:47,297 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:47,300 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:47,300 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,300 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:47,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:47,302 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:47,304 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:47,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:47,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:47,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:47,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:47,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,305 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,304 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:47,305 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,305 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:47,305 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:47,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:47,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:47,405 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:47,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:47,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:47,421 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:47,422 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:47,422 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:47,422 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:47,422 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,422 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:47,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,424 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,424 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:47,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:47,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:47,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:47,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:47,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:47,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:47,427 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:47,427 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:47,427 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:55:47,427 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:47,427 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:47,427 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:47,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:47,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:47,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:47,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:47,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:47,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405807715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405807716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405807726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405807727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405807828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405807828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,838 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/34ba6ba00a994afe924b11ebc119881c 2023-04-25 06:55:47,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405807836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:47,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405807843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,861 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/da5b4d974f81439491b996dc91363ea8 2023-04-25 06:55:47,874 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/36c2b81b1000499985946131bc039043 2023-04-25 06:55:47,889 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/34ba6ba00a994afe924b11ebc119881c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/34ba6ba00a994afe924b11ebc119881c 2023-04-25 06:55:47,892 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/34ba6ba00a994afe924b11ebc119881c, entries=150, sequenceid=146, filesize=11.8 K 2023-04-25 06:55:47,893 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/da5b4d974f81439491b996dc91363ea8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/da5b4d974f81439491b996dc91363ea8 2023-04-25 06:55:47,897 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/da5b4d974f81439491b996dc91363ea8, entries=150, sequenceid=146, filesize=11.8 K 2023-04-25 06:55:47,898 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/36c2b81b1000499985946131bc039043 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c2b81b1000499985946131bc039043 2023-04-25 06:55:47,904 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c2b81b1000499985946131bc039043, entries=150, sequenceid=146, filesize=11.8 K 2023-04-25 06:55:47,905 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 75fc43298851a5c4f1060516f9ac39d4 in 478ms, sequenceid=146, compaction requested=true 2023-04-25 06:55:47,905 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:47,905 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:47,907 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fac957a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:47,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:47,910 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,913 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,913 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:47,913 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:47,913 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:47,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:47,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:47,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:47,914 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:47,914 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:47,917 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:47,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:47,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:47,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:47,917 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:47,918 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,918 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:47,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,918 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:47,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:47,919 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:47,920 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:47,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:47,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:47,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:47,928 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:47,928 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:47,927 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:47,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:47,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:48,028 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:48,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:48,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:48,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:48,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:48,031 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:48,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:48,031 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:48,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,032 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,033 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:48,039 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,040 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,040 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:48,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:48,041 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:55:48,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:48,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:48,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:48,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:48,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:48,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:48,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:48,049 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:48,049 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,049 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:48,052 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:48,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7526843a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:48,055 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:48,055 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,058 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:48,058 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,058 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:48,059 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:48,059 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,059 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,059 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:48,062 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:48,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:48,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:48,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:48,062 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:48,064 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,064 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,064 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,067 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:48,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:48,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,070 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:48,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/59bb8199df4a47038571d303d95dd97d 2023-04-25 06:55:48,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,070 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:48,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/87f9bc542e1448e59bb04992b97a6fa6 2023-04-25 06:55:48,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405808075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405808079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405808086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405808088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,170 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:48,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:48,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:48,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:48,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:48,174 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:48,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:48,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:48,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,174 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,177 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,177 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:48,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:48,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:48,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:48,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,179 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:48,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:48,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:48,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:48,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:48,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:48,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:48,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:48,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,182 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cb5da4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:48,183 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:48,184 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,185 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:48,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:48,186 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:48,186 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,186 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,186 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:48,189 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:48,189 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:48,189 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:48,189 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:48,189 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405808189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405808189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,193 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,193 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,193 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,194 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,195 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,197 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:48,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,198 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:48,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,198 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,198 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:48,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405808199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405808202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,298 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:48,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:48,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:48,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:48,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:48,308 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:48,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:48,308 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:48,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,309 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,310 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,311 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,311 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:48,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:48,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:48,312 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,312 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:48,312 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:48,312 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:48,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:48,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:48,313 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,314 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405808309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,320 DEBUG [Thread-2940] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8199 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:48,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183b628d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:48,322 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:48,322 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,325 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,326 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:48,326 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:48,326 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:48,327 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,327 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:48,327 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:48,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:48,330 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:48,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:48,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:48,330 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:48,330 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:48,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,332 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,332 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,334 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:48,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:48,336 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,336 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:48,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405808393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405808394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405808408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405808409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,437 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:48,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:48,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:48,444 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:48,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:48,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:48,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,447 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,447 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,448 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,448 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:48,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:48,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:48,449 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,449 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:48,449 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:48,449 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:48,456 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@331723b5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:48,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:48,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,462 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:48,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:48,464 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,464 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:48,464 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:48,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:48,466 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:48,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:48,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:48,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:48,466 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:48,467 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,467 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,467 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:48,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,468 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,469 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:48,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:48,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:48,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,473 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:48,474 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:48,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/b1bb3f253ef148e188b1657cd1fadc5c 2023-04-25 06:55:48,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/59bb8199df4a47038571d303d95dd97d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/59bb8199df4a47038571d303d95dd97d 2023-04-25 06:55:48,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/59bb8199df4a47038571d303d95dd97d, entries=200, sequenceid=160, filesize=14.1 K 2023-04-25 06:55:48,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/87f9bc542e1448e59bb04992b97a6fa6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/87f9bc542e1448e59bb04992b97a6fa6 2023-04-25 06:55:48,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/87f9bc542e1448e59bb04992b97a6fa6, entries=150, sequenceid=160, filesize=11.8 K 2023-04-25 06:55:48,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/b1bb3f253ef148e188b1657cd1fadc5c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b1bb3f253ef148e188b1657cd1fadc5c 2023-04-25 06:55:48,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b1bb3f253ef148e188b1657cd1fadc5c, entries=150, sequenceid=160, filesize=11.8 K 2023-04-25 06:55:48,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 75fc43298851a5c4f1060516f9ac39d4 in 479ms, sequenceid=160, compaction requested=true 2023-04-25 06:55:48,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:48,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:48,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:48,520 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 06:55:48,520 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 06:55:48,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:48,523 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85995 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 06:55:48,523 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:55:48,523 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,523 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cfc945f3e1d949ea9f7fd7fbddafaf96, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/44f94e5eaef44081afdec9e25e075d9b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bb5d726da8e4c649dcc0edc65630ee1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bed1d94a8730432fa2d22dec66c40c27, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1b589d13d996451a80c8cdf969c9179b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/34ba6ba00a994afe924b11ebc119881c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/59bb8199df4a47038571d303d95dd97d] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=84.0 K 2023-04-25 06:55:48,523 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 06:55:48,523 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:55:48,523 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,523 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ee49a5d4b3b5463388cdbc6c5aa86ecc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4b09c574f203497a9b7ad554f72108b5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ff560ff0543142a4ab1f3783071692ca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/10d57afc62bd4745ac45d31b9ec67f06, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cabec64139514e8481635a8ba9fdab18, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/da5b4d974f81439491b996dc91363ea8, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/87f9bc542e1448e59bb04992b97a6fa6] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=81.6 K 2023-04-25 06:55:48,524 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting cfc945f3e1d949ea9f7fd7fbddafaf96, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682405741242 2023-04-25 06:55:48,524 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 44f94e5eaef44081afdec9e25e075d9b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682405742394 2023-04-25 06:55:48,525 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 4bb5d726da8e4c649dcc0edc65630ee1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682405743040 2023-04-25 06:55:48,526 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ee49a5d4b3b5463388cdbc6c5aa86ecc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682405741242 2023-04-25 06:55:48,526 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting bed1d94a8730432fa2d22dec66c40c27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1682405744220 2023-04-25 06:55:48,526 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4b09c574f203497a9b7ad554f72108b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682405742394 2023-04-25 06:55:48,527 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 1b589d13d996451a80c8cdf969c9179b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682405745389 2023-04-25 06:55:48,527 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ff560ff0543142a4ab1f3783071692ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682405743040 2023-04-25 06:55:48,527 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 34ba6ba00a994afe924b11ebc119881c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682405746544 2023-04-25 06:55:48,527 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 10d57afc62bd4745ac45d31b9ec67f06, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1682405744220 2023-04-25 06:55:48,528 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 59bb8199df4a47038571d303d95dd97d, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682405747715 2023-04-25 06:55:48,528 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting cabec64139514e8481635a8ba9fdab18, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682405745389 2023-04-25 06:55:48,528 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting da5b4d974f81439491b996dc91363ea8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682405746544 2023-04-25 06:55:48,529 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 87f9bc542e1448e59bb04992b97a6fa6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682405747715 2023-04-25 06:55:48,549 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#392 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:48,559 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#393 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:48,574 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:48,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:48,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:48,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:48,578 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:48,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:48,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:48,579 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,579 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b093d647059a4a9098413fcfcff0bfe2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b093d647059a4a9098413fcfcff0bfe2 2023-04-25 06:55:48,579 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:48,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:48,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:48,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:48,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:48,583 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into b093d647059a4a9098413fcfcff0bfe2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:48,583 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:48,583 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=9, startTime=1682405748520; duration=0sec 2023-04-25 06:55:48,583 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:48,583 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 06:55:48,592 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:48,594 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:48,595 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:48,599 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 06:55:48,599 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:55:48,600 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:48,600 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c3ad46da3d7041c8ae97824615627cba, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c8f90d8c79403685e91f7c3cc522ed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/92a81ce74b8a4cff991f243d5ad25b17, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7ad2169899ee486cb7749b8caf32bf74, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/82a01267182549cd8bd4ed5c0795317b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c2b81b1000499985946131bc039043, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b1bb3f253ef148e188b1657cd1fadc5c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=81.6 K 2023-04-25 06:55:48,600 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting c3ad46da3d7041c8ae97824615627cba, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682405741242 2023-04-25 06:55:48,601 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 36c8f90d8c79403685e91f7c3cc522ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682405742394 2023-04-25 06:55:48,601 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 92a81ce74b8a4cff991f243d5ad25b17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682405743040 2023-04-25 06:55:48,601 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 7ad2169899ee486cb7749b8caf32bf74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1682405744220 2023-04-25 06:55:48,604 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 82a01267182549cd8bd4ed5c0795317b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682405745389 2023-04-25 06:55:48,604 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 36c2b81b1000499985946131bc039043, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682405746544 2023-04-25 06:55:48,605 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b1bb3f253ef148e188b1657cd1fadc5c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682405747715 2023-04-25 06:55:48,619 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#395 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:48,663 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/395450db8b764637897663a4c7583b4a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/395450db8b764637897663a4c7583b4a 2023-04-25 06:55:48,667 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into 395450db8b764637897663a4c7583b4a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:48,667 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:48,667 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=9, startTime=1682405748520; duration=0sec 2023-04-25 06:55:48,667 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:48,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:48,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405808732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405808735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405808740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405808741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405808842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405808846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405808846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:48,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405808849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:48,981 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/8896ba374993461d9f32e70309844450 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8896ba374993461d9f32e70309844450 2023-04-25 06:55:48,987 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into 8896ba374993461d9f32e70309844450(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:48,987 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:48,987 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=9, startTime=1682405748520; duration=0sec 2023-04-25 06:55:48,987 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:49,036 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/58e638452ff34950b57944b72d478b64 2023-04-25 06:55:49,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405809047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405809047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405809056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405809060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405809356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405809356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405809363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405809367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,470 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/5900f70c58e14022afca58bdf816eb9c 2023-04-25 06:55:49,521 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/d899d98eeacc453d8decd9e98110d8e7 2023-04-25 06:55:49,529 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/58e638452ff34950b57944b72d478b64 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/58e638452ff34950b57944b72d478b64 2023-04-25 06:55:49,533 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/58e638452ff34950b57944b72d478b64, entries=150, sequenceid=179, filesize=11.8 K 2023-04-25 06:55:49,533 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/5900f70c58e14022afca58bdf816eb9c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/5900f70c58e14022afca58bdf816eb9c 2023-04-25 06:55:49,554 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/5900f70c58e14022afca58bdf816eb9c, entries=150, sequenceid=179, filesize=11.8 K 2023-04-25 06:55:49,555 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/d899d98eeacc453d8decd9e98110d8e7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d899d98eeacc453d8decd9e98110d8e7 2023-04-25 06:55:49,559 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d899d98eeacc453d8decd9e98110d8e7, entries=150, sequenceid=179, filesize=11.8 K 2023-04-25 06:55:49,559 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 75fc43298851a5c4f1060516f9ac39d4 in 964ms, sequenceid=179, compaction requested=false 2023-04-25 06:55:49,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:49,560 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:49,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725176b9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:49,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:49,564 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,565 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:49,566 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,566 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:49,568 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:49,571 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:49,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:49,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:49,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:49,571 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:49,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:49,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,574 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:49,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,576 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:49,579 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:49,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:49,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:49,580 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:49,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:49,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:49,580 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:49,580 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:49,681 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:49,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:49,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:49,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:49,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:49,697 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:49,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:49,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:49,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:49,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:49,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:49,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:49,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:49,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:49,700 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,700 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:49,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:49,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:49,716 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:49,717 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:49,717 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:49,761 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/37ca9b96bf3142baae15af0009438659 2023-04-25 06:55:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:49,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:49,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405809891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405809898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405809899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:49,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:49,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405809900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405810002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405810012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405810012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405810012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,217 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/fe9297f7e4a646ad92f3c188a883242b 2023-04-25 06:55:50,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405810213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405810222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405810223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405810225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,238 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/02feb1d17f9e4d728ef27c32ada9c467 2023-04-25 06:55:50,244 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/37ca9b96bf3142baae15af0009438659 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37ca9b96bf3142baae15af0009438659 2023-04-25 06:55:50,248 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37ca9b96bf3142baae15af0009438659, entries=150, sequenceid=195, filesize=11.8 K 2023-04-25 06:55:50,249 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/fe9297f7e4a646ad92f3c188a883242b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/fe9297f7e4a646ad92f3c188a883242b 2023-04-25 06:55:50,265 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/fe9297f7e4a646ad92f3c188a883242b, entries=150, sequenceid=195, filesize=11.8 K 2023-04-25 06:55:50,266 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/02feb1d17f9e4d728ef27c32ada9c467 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/02feb1d17f9e4d728ef27c32ada9c467 2023-04-25 06:55:50,277 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/02feb1d17f9e4d728ef27c32ada9c467, entries=150, sequenceid=195, filesize=11.8 K 2023-04-25 06:55:50,278 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 75fc43298851a5c4f1060516f9ac39d4 in 561ms, sequenceid=195, compaction requested=true 2023-04-25 06:55:50,278 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:50,278 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:50,282 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e2b232a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:50,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:50,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,288 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:50,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:50,290 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:50,293 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:50,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:50,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:50,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:50,293 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:50,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,296 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:50,300 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,303 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:50,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:50,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:50,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:50,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:50,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:50,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:50,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,307 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:50,307 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:50,308 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:50,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:50,408 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:50,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:50,413 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:50,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:50,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:50,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:50,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,413 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:50,414 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:50,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:50,414 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,415 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,415 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:50,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:50,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:50,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:50,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:50,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:50,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:50,428 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:55:50,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:50,430 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:50,430 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:50,430 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:50,430 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:50,430 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:50,472 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/28270bef748a4eb5af0918d77e632073 2023-04-25 06:55:50,522 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/9f2a07b8b43847b68f74c15d50d6d200 2023-04-25 06:55:50,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:50,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:50,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405810565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405810567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405810574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56488 deadline: 1682405810574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56530 deadline: 1682405810679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405810682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405810688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405810689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405810886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56528 deadline: 1682405810886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405810898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56540 deadline: 1682405810898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,950 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/a5891719067d4954be47149921ae4512 2023-04-25 06:55:50,963 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/28270bef748a4eb5af0918d77e632073 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/28270bef748a4eb5af0918d77e632073 2023-04-25 06:55:50,967 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/28270bef748a4eb5af0918d77e632073, entries=150, sequenceid=213, filesize=11.8 K 2023-04-25 06:55:50,968 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/9f2a07b8b43847b68f74c15d50d6d200 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9f2a07b8b43847b68f74c15d50d6d200 2023-04-25 06:55:50,972 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9f2a07b8b43847b68f74c15d50d6d200, entries=150, sequenceid=213, filesize=11.8 K 2023-04-25 06:55:50,973 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/a5891719067d4954be47149921ae4512 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a5891719067d4954be47149921ae4512 2023-04-25 06:55:50,976 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a5891719067d4954be47149921ae4512, entries=150, sequenceid=213, filesize=11.8 K 2023-04-25 06:55:50,977 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 75fc43298851a5c4f1060516f9ac39d4 in 549ms, sequenceid=213, compaction requested=true 2023-04-25 06:55:50,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:50,977 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:50,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:50,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:50,980 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:50,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc70368[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:50,983 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,985 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:50,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:50,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:50,986 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,986 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:50,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:50,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:50,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:50,987 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:50,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:50,990 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:50,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:50,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:50,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:50,990 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:50,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:50,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:50,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:50,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:50,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:50,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:50,998 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:50,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:50,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:50,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:50,999 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:50,999 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:50,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:50,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:51,099 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:51,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:51,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:51,120 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:51,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:51,120 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:51,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,123 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,123 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:51,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:51,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:51,133 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:51,182 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/340ca07384e745ea9af24801de2b85d6 2023-04-25 06:55:51,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:51,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:51,214 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/4494c6d2b0a5418cac653f643689c778 2023-04-25 06:55:51,234 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/9764ecf3a16d40c1b8ef41ed83841abb 2023-04-25 06:55:51,238 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/340ca07384e745ea9af24801de2b85d6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/340ca07384e745ea9af24801de2b85d6 2023-04-25 06:55:51,241 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/340ca07384e745ea9af24801de2b85d6, entries=150, sequenceid=228, filesize=11.8 K 2023-04-25 06:55:51,241 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/4494c6d2b0a5418cac653f643689c778 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4494c6d2b0a5418cac653f643689c778 2023-04-25 06:55:51,245 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4494c6d2b0a5418cac653f643689c778, entries=150, sequenceid=228, filesize=11.8 K 2023-04-25 06:55:51,245 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/9764ecf3a16d40c1b8ef41ed83841abb as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9764ecf3a16d40c1b8ef41ed83841abb 2023-04-25 06:55:51,255 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9764ecf3a16d40c1b8ef41ed83841abb, entries=150, sequenceid=228, filesize=11.8 K 2023-04-25 06:55:51,256 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=80.51 KB/82440 for 75fc43298851a5c4f1060516f9ac39d4 in 123ms, sequenceid=228, compaction requested=true 2023-04-25 06:55:51,256 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:51,256 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,256 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:55:51,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:51,257 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:51,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:51,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:51,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:51,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:51,258 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,259 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,259 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:51,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67479fee[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:51,260 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:51,261 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:51,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:51,263 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:51,265 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:51,265 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:51,265 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:51,266 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:51,266 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:51,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,275 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:51,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:51,275 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:51,275 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:51,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:51,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405811304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405811309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405811319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405811319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,375 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:51,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:51,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:51,379 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:51,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:51,379 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:51,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,380 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,381 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,381 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:51,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:51,383 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,383 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:51,383 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:51,383 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:51,384 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5ac901[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:51,387 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:51,387 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,389 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:51,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,390 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,391 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:51,395 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:51,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:51,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:51,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:51,395 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:51,396 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,396 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,396 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,399 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,400 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:51,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:51,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,403 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:51,403 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:51,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,403 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405811420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405811420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405811436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405811440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,503 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:51,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:51,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:51,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:51,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:51,511 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:51,511 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,511 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:51,511 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:51,512 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,512 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,514 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,514 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,515 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:51,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:51,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:51,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:51,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:51,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:51,522 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691f9b93[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:51,524 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:51,524 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,525 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:51,525 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,525 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:51,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:51,529 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:51,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:51,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:51,529 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:51,529 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:51,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,534 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,535 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,540 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:51,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:51,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,540 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:51,540 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:51,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405811629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,640 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:51,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:51,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405811630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:51,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405811641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:51,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:51,645 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:51,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:51,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:51,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,647 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:51,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:51,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:51,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:51,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:51,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:51,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:51,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:51,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,655 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:51,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405811645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c16199b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:51,658 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:51,658 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,659 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:51,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,660 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:51,662 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:51,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:51,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:51,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:51,662 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:51,665 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,665 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,665 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,672 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:51,673 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,673 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:51,673 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:51,673 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:51,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/c0d0e2d4977d4e85ab4375ad74617d03 2023-04-25 06:55:51,773 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:51,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:51,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:51,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:51,804 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:51,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:51,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:51,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,806 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:51,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:51,816 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,816 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:51,817 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:51,817 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:51,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:51,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f54918[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:51,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:51,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:51,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:51,826 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:51,829 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:51,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:51,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:51,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:51,829 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:51,831 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,831 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,832 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,835 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:51,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,839 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:51,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,840 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,840 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:51,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:51,940 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:51,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:51,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:51,943 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:51,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:51,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:51,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,945 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,945 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405811936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:51,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:51,948 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,948 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:51,948 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:51,948 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:51,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405811943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405811946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:51,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:51,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2faf9626[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:51,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,958 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:51,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,959 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:51,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,961 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:51,961 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,961 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:51,963 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:51,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:51,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:51,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:51,963 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:51,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:51,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405811956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,966 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:51,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,969 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:51,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:51,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:51,978 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:51,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:51,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:51,979 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:51,978 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:51,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:52,079 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:52,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:52,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:52,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:52,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:52,082 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:52,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:52,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:52,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:52,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:52,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:52,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:52,086 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:52,088 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:52,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:52,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:52,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:52,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a795343[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:52,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:52,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:52,091 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,091 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:52,093 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:52,103 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:52,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:52,104 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:52,104 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:52,104 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:52,106 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,106 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:52,106 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:52,108 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:52,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:52,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/9bb1afb1ed644477a4a9d6bb957560fa 2023-04-25 06:55:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,112 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:52,112 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,112 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:52,112 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:52,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:52,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:52,112 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:52,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,113 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:52,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:52,113 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:52,124 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/14817c7a27c94c22b5955768d2333d68 2023-04-25 06:55:52,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/c0d0e2d4977d4e85ab4375ad74617d03 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/c0d0e2d4977d4e85ab4375ad74617d03 2023-04-25 06:55:52,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/c0d0e2d4977d4e85ab4375ad74617d03, entries=300, sequenceid=245, filesize=18.8 K 2023-04-25 06:55:52,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/9bb1afb1ed644477a4a9d6bb957560fa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9bb1afb1ed644477a4a9d6bb957560fa 2023-04-25 06:55:52,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9bb1afb1ed644477a4a9d6bb957560fa, entries=150, sequenceid=245, filesize=11.8 K 2023-04-25 06:55:52,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/14817c7a27c94c22b5955768d2333d68 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/14817c7a27c94c22b5955768d2333d68 2023-04-25 06:55:52,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/14817c7a27c94c22b5955768d2333d68, entries=150, sequenceid=245, filesize=11.8 K 2023-04-25 06:55:52,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 75fc43298851a5c4f1060516f9ac39d4 in 888ms, sequenceid=245, compaction requested=true 2023-04-25 06:55:52,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:52,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:52,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:52,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:55:52,144 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:55:52,144 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:55:52,146 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72456 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:55:52,146 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 79626 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:55:52,146 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:55:52,146 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:55:52,146 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:52,146 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:52,146 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8896ba374993461d9f32e70309844450, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/5900f70c58e14022afca58bdf816eb9c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/fe9297f7e4a646ad92f3c188a883242b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9f2a07b8b43847b68f74c15d50d6d200, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4494c6d2b0a5418cac653f643689c778, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9bb1afb1ed644477a4a9d6bb957560fa] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=70.8 K 2023-04-25 06:55:52,147 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b093d647059a4a9098413fcfcff0bfe2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/58e638452ff34950b57944b72d478b64, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37ca9b96bf3142baae15af0009438659, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/28270bef748a4eb5af0918d77e632073, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/340ca07384e745ea9af24801de2b85d6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/c0d0e2d4977d4e85ab4375ad74617d03] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=77.8 K 2023-04-25 06:55:52,147 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 8896ba374993461d9f32e70309844450, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682405747715 2023-04-25 06:55:52,147 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b093d647059a4a9098413fcfcff0bfe2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682405747715 2023-04-25 06:55:52,147 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5900f70c58e14022afca58bdf816eb9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682405748069 2023-04-25 06:55:52,147 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 58e638452ff34950b57944b72d478b64, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682405748069 2023-04-25 06:55:52,148 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting fe9297f7e4a646ad92f3c188a883242b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682405748715 2023-04-25 06:55:52,148 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 37ca9b96bf3142baae15af0009438659, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682405748715 2023-04-25 06:55:52,148 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9f2a07b8b43847b68f74c15d50d6d200, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682405749886 2023-04-25 06:55:52,148 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 28270bef748a4eb5af0918d77e632073, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682405749886 2023-04-25 06:55:52,148 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 4494c6d2b0a5418cac653f643689c778, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405750544 2023-04-25 06:55:52,149 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 340ca07384e745ea9af24801de2b85d6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405750544 2023-04-25 06:55:52,149 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9bb1afb1ed644477a4a9d6bb957560fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682405751254 2023-04-25 06:55:52,149 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c0d0e2d4977d4e85ab4375ad74617d03, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682405751219 2023-04-25 06:55:52,162 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#410 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:52,175 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#411 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:52,184 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/340394ee2e6647c68ba49adebffb68f5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/340394ee2e6647c68ba49adebffb68f5 2023-04-25 06:55:52,188 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into 340394ee2e6647c68ba49adebffb68f5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:52,188 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:52,188 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=10, startTime=1682405752144; duration=0sec 2023-04-25 06:55:52,188 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:52,188 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:55:52,191 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72456 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:55:52,191 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:55:52,191 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:52,191 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/395450db8b764637897663a4c7583b4a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d899d98eeacc453d8decd9e98110d8e7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/02feb1d17f9e4d728ef27c32ada9c467, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a5891719067d4954be47149921ae4512, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9764ecf3a16d40c1b8ef41ed83841abb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/14817c7a27c94c22b5955768d2333d68] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=70.8 K 2023-04-25 06:55:52,192 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 395450db8b764637897663a4c7583b4a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682405747715 2023-04-25 06:55:52,192 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d899d98eeacc453d8decd9e98110d8e7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682405748069 2023-04-25 06:55:52,192 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 02feb1d17f9e4d728ef27c32ada9c467, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682405748715 2023-04-25 06:55:52,193 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a5891719067d4954be47149921ae4512, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682405749886 2023-04-25 06:55:52,193 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9764ecf3a16d40c1b8ef41ed83841abb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1682405750544 2023-04-25 06:55:52,193 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 14817c7a27c94c22b5955768d2333d68, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682405751254 2023-04-25 06:55:52,203 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#412 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:52,214 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:52,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:52,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:52,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:52,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:52,218 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:52,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:52,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:52,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:52,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,220 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:52,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:52,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:52,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:52,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:52,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:52,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:52,225 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:52,225 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:52,225 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 06:55:52,225 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:52,225 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:52,225 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:52,225 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:52,225 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:52,226 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:52,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:52,450 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:52,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405812483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405812484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405812485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405812491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,596 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/d97868e8661f41d3b0fb475a97189aee as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/d97868e8661f41d3b0fb475a97189aee 2023-04-25 06:55:52,600 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into d97868e8661f41d3b0fb475a97189aee(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:52,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,600 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:52,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405812593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,600 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=10, startTime=1682405752144; duration=0sec 2023-04-25 06:55:52,600 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:52,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405812595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405812595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405812595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,616 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/7b771d3cb37d49e79ed35170e66edc2b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7b771d3cb37d49e79ed35170e66edc2b 2023-04-25 06:55:52,620 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into 7b771d3cb37d49e79ed35170e66edc2b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:52,620 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:52,620 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=10, startTime=1682405752144; duration=0sec 2023-04-25 06:55:52,620 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:52,647 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/cb70995b592e4ddc9974c0aa892eeec9 2023-04-25 06:55:52,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405812801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405812802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405812803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:52,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405812803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,059 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/47e13b17b72d4eb69e7a489e35a3ccdf 2023-04-25 06:55:53,071 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/4e89ec1ca9b7480dbfec9454d5882bad 2023-04-25 06:55:53,075 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/cb70995b592e4ddc9974c0aa892eeec9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cb70995b592e4ddc9974c0aa892eeec9 2023-04-25 06:55:53,078 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cb70995b592e4ddc9974c0aa892eeec9, entries=150, sequenceid=262, filesize=11.9 K 2023-04-25 06:55:53,079 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/47e13b17b72d4eb69e7a489e35a3ccdf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47e13b17b72d4eb69e7a489e35a3ccdf 2023-04-25 06:55:53,082 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47e13b17b72d4eb69e7a489e35a3ccdf, entries=150, sequenceid=262, filesize=11.9 K 2023-04-25 06:55:53,087 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/4e89ec1ca9b7480dbfec9454d5882bad as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/4e89ec1ca9b7480dbfec9454d5882bad 2023-04-25 06:55:53,091 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/4e89ec1ca9b7480dbfec9454d5882bad, entries=150, sequenceid=262, filesize=11.9 K 2023-04-25 06:55:53,092 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 75fc43298851a5c4f1060516f9ac39d4 in 867ms, sequenceid=262, compaction requested=false 2023-04-25 06:55:53,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:53,092 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:53,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:53,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:53,092 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:53,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@362f4f56[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:53,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:53,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,100 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:53,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:53,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:53,102 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:53,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:53,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:53,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:53,102 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:53,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,104 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,105 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:53,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:55:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,113 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,114 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:53,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:53,114 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:53,114 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:53,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/5d5e05f7a44a40d1b3e352250ebf668b 2023-04-25 06:55:53,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405813135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405813136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405813141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405813141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/05e8c0e3a24b40409775a19b46b77d32 2023-04-25 06:55:53,214 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:53,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:53,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:53,218 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:53,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:53,221 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,221 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:53,221 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:53,221 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:53,223 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b5f6d02[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:53,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:53,225 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:53,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:53,228 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,229 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:53,229 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,229 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:53,231 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:53,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:53,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:53,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:53,231 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:53,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,235 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,241 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:53,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,241 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:53,241 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:53,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405813243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405813243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405813244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405813244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,341 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:53,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:53,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:53,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:53,345 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:53,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:53,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:53,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:53,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:53,352 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,352 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:53,352 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:53,352 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:53,353 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2812dae9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:53,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:53,355 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,356 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:53,357 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,357 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:53,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:53,362 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:53,362 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:53,362 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:53,362 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:53,362 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:53,364 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,364 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,364 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,364 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,367 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:53,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:53,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:53,368 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,368 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,368 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,368 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,368 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,368 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:53,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,368 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,368 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:53,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405813454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405813454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405813455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405813455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,468 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:53,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:53,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:53,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:53,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:53,471 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:53,472 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,472 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:53,472 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:53,472 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,472 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,474 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,474 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:53,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:53,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:53,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:53,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:53,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:53,478 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@580177e3[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:53,480 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:53,480 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,481 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:53,482 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,482 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:53,484 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:53,486 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:53,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:53,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:53,486 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:53,486 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:53,488 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,488 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:53,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,488 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:53,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,488 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,490 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,493 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:53,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:53,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:53,493 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,493 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:53,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,493 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,494 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:53,493 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:53,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:53,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:53,494 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/546ae7f060e74cc98c94270b7128d9c7 2023-04-25 06:55:53,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/5d5e05f7a44a40d1b3e352250ebf668b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/5d5e05f7a44a40d1b3e352250ebf668b 2023-04-25 06:55:53,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/5d5e05f7a44a40d1b3e352250ebf668b, entries=250, sequenceid=282, filesize=16.7 K 2023-04-25 06:55:53,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/05e8c0e3a24b40409775a19b46b77d32 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/05e8c0e3a24b40409775a19b46b77d32 2023-04-25 06:55:53,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/05e8c0e3a24b40409775a19b46b77d32, entries=150, sequenceid=282, filesize=11.9 K 2023-04-25 06:55:53,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/546ae7f060e74cc98c94270b7128d9c7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/546ae7f060e74cc98c94270b7128d9c7 2023-04-25 06:55:53,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/546ae7f060e74cc98c94270b7128d9c7, entries=150, sequenceid=282, filesize=11.9 K 2023-04-25 06:55:53,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 75fc43298851a5c4f1060516f9ac39d4 in 474ms, sequenceid=282, compaction requested=true 2023-04-25 06:55:53,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:53,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:53,585 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:53,585 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:53,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:53,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:53,586 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:53,586 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:55:53,586 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,587 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/d97868e8661f41d3b0fb475a97189aee, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cb70995b592e4ddc9974c0aa892eeec9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/5d5e05f7a44a40d1b3e352250ebf668b] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=40.7 K 2023-04-25 06:55:53,587 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36802 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:53,587 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:55:53,587 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,587 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting d97868e8661f41d3b0fb475a97189aee, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682405751254 2023-04-25 06:55:53,587 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/340394ee2e6647c68ba49adebffb68f5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47e13b17b72d4eb69e7a489e35a3ccdf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/05e8c0e3a24b40409775a19b46b77d32] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=35.9 K 2023-04-25 06:55:53,587 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting cb70995b592e4ddc9974c0aa892eeec9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682405751301 2023-04-25 06:55:53,587 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 340394ee2e6647c68ba49adebffb68f5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682405751254 2023-04-25 06:55:53,587 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 5d5e05f7a44a40d1b3e352250ebf668b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1682405752482 2023-04-25 06:55:53,588 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 47e13b17b72d4eb69e7a489e35a3ccdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682405751301 2023-04-25 06:55:53,588 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 05e8c0e3a24b40409775a19b46b77d32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1682405752482 2023-04-25 06:55:53,594 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:53,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:53,594 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:53,596 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:53,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:53,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:53,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:53,599 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:53,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:53,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:53,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,599 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,600 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,601 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:53,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:53,601 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,601 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:53,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:53,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:53,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:53,604 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,605 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:53,605 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:55:53,605 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:53,613 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/f1126ad0f8bc4a97a2bc2f7bfa5ac3b3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/f1126ad0f8bc4a97a2bc2f7bfa5ac3b3 2023-04-25 06:55:53,621 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into f1126ad0f8bc4a97a2bc2f7bfa5ac3b3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:53,621 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:53,621 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=13, startTime=1682405753585; duration=0sec 2023-04-25 06:55:53,621 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:53,621 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:53,626 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36802 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:53,626 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:55:53,626 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:53,626 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7b771d3cb37d49e79ed35170e66edc2b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/4e89ec1ca9b7480dbfec9454d5882bad, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/546ae7f060e74cc98c94270b7128d9c7] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=35.9 K 2023-04-25 06:55:53,626 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7b771d3cb37d49e79ed35170e66edc2b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682405751254 2023-04-25 06:55:53,627 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b9eda3d2aae246bab3b79e92ebb8d53d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b9eda3d2aae246bab3b79e92ebb8d53d 2023-04-25 06:55:53,627 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4e89ec1ca9b7480dbfec9454d5882bad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682405751301 2023-04-25 06:55:53,627 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 546ae7f060e74cc98c94270b7128d9c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1682405752482 2023-04-25 06:55:53,635 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into b9eda3d2aae246bab3b79e92ebb8d53d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:53,635 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:53,635 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=13, startTime=1682405753585; duration=0sec 2023-04-25 06:55:53,635 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:53,648 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#422 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:53,669 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/349a0471e5574d6484d1854a0d9371f1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/349a0471e5574d6484d1854a0d9371f1 2023-04-25 06:55:53,674 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into 349a0471e5574d6484d1854a0d9371f1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:53,674 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:53,674 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=13, startTime=1682405753585; duration=0sec 2023-04-25 06:55:53,674 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:53,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:53,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:53,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405813776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405813780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405813782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405813783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405813884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405813884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405813891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:53,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:53,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405813891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,037 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b5ad1b35e54c4e3f80b230558af9aab1 2023-04-25 06:55:54,048 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/623d117b10024b3aa74c61e1d9f0a49d 2023-04-25 06:55:54,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405814090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405814092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405814096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405814096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405814395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405814395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405814400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405814401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,476 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/122d347388cb48c1af8d69e24c1cc362 2023-04-25 06:55:54,480 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b5ad1b35e54c4e3f80b230558af9aab1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b5ad1b35e54c4e3f80b230558af9aab1 2023-04-25 06:55:54,482 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b5ad1b35e54c4e3f80b230558af9aab1, entries=150, sequenceid=297, filesize=11.9 K 2023-04-25 06:55:54,483 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/623d117b10024b3aa74c61e1d9f0a49d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/623d117b10024b3aa74c61e1d9f0a49d 2023-04-25 06:55:54,487 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/623d117b10024b3aa74c61e1d9f0a49d, entries=150, sequenceid=297, filesize=11.9 K 2023-04-25 06:55:54,488 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/122d347388cb48c1af8d69e24c1cc362 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/122d347388cb48c1af8d69e24c1cc362 2023-04-25 06:55:54,491 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/122d347388cb48c1af8d69e24c1cc362, entries=150, sequenceid=297, filesize=11.9 K 2023-04-25 06:55:54,492 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 75fc43298851a5c4f1060516f9ac39d4 in 886ms, sequenceid=297, compaction requested=false 2023-04-25 06:55:54,492 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:54,492 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:54,493 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,494 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:54,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1498c64a[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:54,495 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:54,495 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,496 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,496 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:54,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:54,496 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:54,497 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:54,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:54,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,499 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,499 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:54,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:54,501 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:54,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:54,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:54,501 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:54,501 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:54,503 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,503 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,503 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,504 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,505 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:54,507 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:54,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:54,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:54,507 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:54,508 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:54,508 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:54,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:54,608 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:54,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:54,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:54,611 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:54,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:54,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:54,611 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:54,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,612 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,613 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:54,613 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:54,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:54,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:54,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:54,615 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:54,626 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/612d71e9c9604d9eb4bd9e037260c361 2023-04-25 06:55:54,637 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/68861ec9f110418488007c111d0765f7 2023-04-25 06:55:54,651 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/ff6a745658174b229de2605d3b40aadc 2023-04-25 06:55:54,658 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/612d71e9c9604d9eb4bd9e037260c361 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/612d71e9c9604d9eb4bd9e037260c361 2023-04-25 06:55:54,663 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/612d71e9c9604d9eb4bd9e037260c361, entries=150, sequenceid=317, filesize=11.9 K 2023-04-25 06:55:54,663 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/68861ec9f110418488007c111d0765f7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/68861ec9f110418488007c111d0765f7 2023-04-25 06:55:54,667 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/68861ec9f110418488007c111d0765f7, entries=150, sequenceid=317, filesize=11.9 K 2023-04-25 06:55:54,668 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/ff6a745658174b229de2605d3b40aadc as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/ff6a745658174b229de2605d3b40aadc 2023-04-25 06:55:54,677 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/ff6a745658174b229de2605d3b40aadc, entries=150, sequenceid=317, filesize=11.9 K 2023-04-25 06:55:54,677 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 75fc43298851a5c4f1060516f9ac39d4 in 62ms, sequenceid=317, compaction requested=true 2023-04-25 06:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:54,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:54,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:54,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:54,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:54,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142f8ea6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:54,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:54,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:54,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:54,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:54,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:54,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:54,689 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:54,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:54,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:54,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:54,690 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:54,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,693 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:54,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:54,697 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:54,698 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:54,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,698 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:54,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:54,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,798 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:54,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:54,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:54,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:54,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:54,814 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:54,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:54,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:54,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:54,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,816 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:54,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,817 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,817 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:54,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:54,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:54,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:54,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:54,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:54,825 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:54,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:54,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,826 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a54cb58[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:54,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:54,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,830 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:54,830 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,830 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:54,833 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:54,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:54,835 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:54,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:54,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:54,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:54,836 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:54,837 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,837 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,837 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,838 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:54,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:54,841 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,842 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:54,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:54,842 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:54,842 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:54,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:54,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:55:54,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:54,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:54,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:54,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:54,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:54,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:54,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/2558aab9aff744518fe55ffe9be4877f 2023-04-25 06:55:54,943 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:54,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:54,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:54,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:54,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:54,946 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:54,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:54,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:54,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:54,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:54,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:54,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:54,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:54,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:54,950 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:54,950 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:54,950 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:54,950 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:54,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79391f9e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:54,954 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:54,955 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:54,956 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,956 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:54,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:54,963 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:54,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:54,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:54,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:54,963 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:54,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:54,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:54,965 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,969 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:54,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:54,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:54,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:54,969 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:54,969 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:54,970 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:54,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:54,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:54,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:54,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405814965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:54,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405754944, max timestamp=1682405754974Num uniques 859; 2023-04-25 06:55:54,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:54,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405754944, max timestamp=1682405754974Num uniques 859; 2023-04-25 06:55:54,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:54,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405754944, max timestamp=1682405754974Num uniques 859; 2023-04-25 06:55:54,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:55:55,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405815008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405815010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405815009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,070 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:55,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:55,073 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:55,073 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,073 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:55,073 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:55,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,075 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:55,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:55,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:55,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:55,077 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:55,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:55,079 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b413c65[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:55,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:55,081 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405815076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,085 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:55,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:55,087 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:55,090 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:55,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:55,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:55,090 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:55,090 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:55,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,092 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,093 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,095 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,095 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,096 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:55,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:55,096 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,097 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:55,097 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:55,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:55,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405815118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405815119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405815119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,207 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:55,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:55,211 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:55,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:55,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:55,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,212 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,213 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,213 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:55,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:55,215 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,215 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:55,215 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:55,215 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:55,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,217 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:55,218 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f3dd15[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:55,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:55,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:55,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:55,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:55,223 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:55,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:55,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:55,223 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:55,223 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:55,225 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,225 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,225 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,226 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,227 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:55,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:55,229 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:55,229 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,229 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:55,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405815283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,329 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:55,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:55,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:55,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:55,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:55,333 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:55,333 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,333 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:55,333 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:55,334 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,334 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405815324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405815324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405815324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,339 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,340 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:55,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:55,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:55,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:55,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:55,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:55,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:55,345 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f71c038[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:55,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:55,348 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:55,351 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:55,351 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,351 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:55,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:55,352 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,353 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/0ca7decfedab42cdbbd10c11565f4959 2023-04-25 06:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:55,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:55,358 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:55,358 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:55,358 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:55,358 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:55,358 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:55,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,361 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,362 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,365 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/011848a908a840cbbbfe7da39cb0a9f5 2023-04-25 06:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:55,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,366 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,366 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:55,367 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:55,366 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,367 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/2558aab9aff744518fe55ffe9be4877f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/2558aab9aff744518fe55ffe9be4877f 2023-04-25 06:55:55,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/2558aab9aff744518fe55ffe9be4877f, entries=250, sequenceid=329, filesize=16.7 K 2023-04-25 06:55:55,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/0ca7decfedab42cdbbd10c11565f4959 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/0ca7decfedab42cdbbd10c11565f4959 2023-04-25 06:55:55,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/0ca7decfedab42cdbbd10c11565f4959, entries=150, sequenceid=329, filesize=11.9 K 2023-04-25 06:55:55,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/011848a908a840cbbbfe7da39cb0a9f5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/011848a908a840cbbbfe7da39cb0a9f5 2023-04-25 06:55:55,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/011848a908a840cbbbfe7da39cb0a9f5, entries=150, sequenceid=329, filesize=11.9 K 2023-04-25 06:55:55,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 75fc43298851a5c4f1060516f9ac39d4 in 458ms, sequenceid=329, compaction requested=true 2023-04-25 06:55:55,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:55,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:55,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:55,381 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:55,381 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:55,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:55,383 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54170 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:55,383 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:55,383 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:55:55,383 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:55:55,383 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,383 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,383 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b9eda3d2aae246bab3b79e92ebb8d53d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b5ad1b35e54c4e3f80b230558af9aab1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/612d71e9c9604d9eb4bd9e037260c361, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/2558aab9aff744518fe55ffe9be4877f] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=52.9 K 2023-04-25 06:55:55,383 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/f1126ad0f8bc4a97a2bc2f7bfa5ac3b3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/623d117b10024b3aa74c61e1d9f0a49d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/68861ec9f110418488007c111d0765f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/0ca7decfedab42cdbbd10c11565f4959] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=48.1 K 2023-04-25 06:55:55,383 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b9eda3d2aae246bab3b79e92ebb8d53d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1682405752482 2023-04-25 06:55:55,383 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting f1126ad0f8bc4a97a2bc2f7bfa5ac3b3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1682405752482 2023-04-25 06:55:55,384 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b5ad1b35e54c4e3f80b230558af9aab1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682405753135 2023-04-25 06:55:55,384 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 623d117b10024b3aa74c61e1d9f0a49d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682405753135 2023-04-25 06:55:55,385 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 612d71e9c9604d9eb4bd9e037260c361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682405753774 2023-04-25 06:55:55,386 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 68861ec9f110418488007c111d0765f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682405753774 2023-04-25 06:55:55,386 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 2558aab9aff744518fe55ffe9be4877f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682405754909 2023-04-25 06:55:55,386 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 0ca7decfedab42cdbbd10c11565f4959, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682405754916 2023-04-25 06:55:55,395 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#431 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:55,397 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#432 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:55,410 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/011d49824b1747b196b09fdec8561280 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/011d49824b1747b196b09fdec8561280 2023-04-25 06:55:55,412 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/66f0f6c5ac9547f5b14133c649f898dd as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/66f0f6c5ac9547f5b14133c649f898dd 2023-04-25 06:55:55,414 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into 011d49824b1747b196b09fdec8561280(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:55,414 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:55,414 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=12, startTime=1682405755381; duration=0sec 2023-04-25 06:55:55,414 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:55,415 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:55:55,416 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:55:55,416 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:55:55,416 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,416 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/349a0471e5574d6484d1854a0d9371f1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/122d347388cb48c1af8d69e24c1cc362, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/ff6a745658174b229de2605d3b40aadc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/011848a908a840cbbbfe7da39cb0a9f5] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=48.1 K 2023-04-25 06:55:55,416 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 349a0471e5574d6484d1854a0d9371f1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1682405752482 2023-04-25 06:55:55,416 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into 66f0f6c5ac9547f5b14133c649f898dd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:55,417 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:55,417 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=12, startTime=1682405755381; duration=0sec 2023-04-25 06:55:55,417 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:55,417 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 122d347388cb48c1af8d69e24c1cc362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682405753135 2023-04-25 06:55:55,417 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ff6a745658174b229de2605d3b40aadc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682405753774 2023-04-25 06:55:55,417 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 011848a908a840cbbbfe7da39cb0a9f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682405754916 2023-04-25 06:55:55,424 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#433 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:55,431 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/b199729201c241a59005aeef8247bc62 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b199729201c241a59005aeef8247bc62 2023-04-25 06:55:55,435 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into b199729201c241a59005aeef8247bc62(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:55,435 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:55,435 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=12, startTime=1682405755381; duration=0sec 2023-04-25 06:55:55,435 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:55,467 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:55,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:55,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:55,470 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:55,470 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,470 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:55,470 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:55,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,473 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:55,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:55,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:55,475 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,475 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:55,475 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 06:55:55,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:55,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:55,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:55,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:55,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:55,476 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:55:55,489 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/1bd18ff00e104bc58ab27b3f13c50a16 2023-04-25 06:55:55,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:55,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:55,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405815637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405815637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405815639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405815640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:55,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405815743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,904 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/8dc9ee34ed7b4459be23dc04fd687bfa 2023-04-25 06:55:55,914 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/538b79639c2240ec837cf5b771d8dc8d 2023-04-25 06:55:55,918 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/1bd18ff00e104bc58ab27b3f13c50a16 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1bd18ff00e104bc58ab27b3f13c50a16 2023-04-25 06:55:55,922 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1bd18ff00e104bc58ab27b3f13c50a16, entries=150, sequenceid=366, filesize=11.9 K 2023-04-25 06:55:55,922 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/8dc9ee34ed7b4459be23dc04fd687bfa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8dc9ee34ed7b4459be23dc04fd687bfa 2023-04-25 06:55:55,930 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8dc9ee34ed7b4459be23dc04fd687bfa, entries=150, sequenceid=366, filesize=11.9 K 2023-04-25 06:55:55,931 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/538b79639c2240ec837cf5b771d8dc8d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/538b79639c2240ec837cf5b771d8dc8d 2023-04-25 06:55:55,935 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/538b79639c2240ec837cf5b771d8dc8d, entries=150, sequenceid=366, filesize=11.9 K 2023-04-25 06:55:55,936 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 75fc43298851a5c4f1060516f9ac39d4 in 461ms, sequenceid=366, compaction requested=false 2023-04-25 06:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:55,947 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:55,947 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,947 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:55,947 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:55,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:55,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:55:55,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:55,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:55,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:55,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:55,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:55,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e85e3a7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:55,960 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:55,960 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,962 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:55,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:55,962 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:55,962 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,963 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:55,971 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:55,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,971 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:55,975 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:55,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:55,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:55,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:55,975 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:55,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:55,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,977 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,977 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:55,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:55,978 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,980 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,980 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,980 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:55,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:55,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:55,980 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:55,981 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:55,981 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,981 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:55,981 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:55,981 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:55,981 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:55,981 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:55,981 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:55,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:55,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:55,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:56,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:56,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405816058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,081 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:56,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:56,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:56,084 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:56,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:56,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:56,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,085 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,086 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:56,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:56,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:56,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:56,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:56,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:56,087 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:56,087 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,087 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:56,088 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:56,088 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:56,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:56,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:56,088 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:56,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:56,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c055a06[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:56,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:56,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,093 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:56,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:56,093 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:56,093 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,093 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:56,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:56,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,094 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:56,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:56,097 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:56,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:56,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:56,097 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:56,097 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:56,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,098 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,099 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,100 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,103 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,104 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,104 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:56,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:56,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,104 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,104 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:56,104 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:56,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:56,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405816143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:56,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405816144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:56,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405816147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405816165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,204 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:56,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:56,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:56,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:56,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:56,208 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:56,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:56,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:56,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,209 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:56,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,211 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:56,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:56,212 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,212 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:56,212 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:56,212 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:56,213 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,213 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,214 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,214 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:56,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1433fe6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:56,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:56,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,218 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:56,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,219 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:56,221 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:56,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,221 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:56,224 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:56,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:56,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:56,224 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:56,224 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:56,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,226 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,227 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,228 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,230 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,233 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:56,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:56,233 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:56,233 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:56,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:56,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,334 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:56,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:56,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:56,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:56,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:56,337 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:56,337 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,337 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:56,337 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:56,337 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,337 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,339 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:56,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:56,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:56,340 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,340 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:56,340 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:56,340 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:56,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:56,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,343 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:56,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40f24ff6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:56,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:56,345 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,346 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:56,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:56,346 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:56,346 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,347 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:56,348 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,348 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:56,348 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,348 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:56,350 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:56,350 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:56,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,352 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,353 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,354 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:56,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:56,360 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,360 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,361 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:56,361 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:56,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/014f8cf887184deb833ebad064e6a669 2023-04-25 06:55:56,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:56,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405816371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,461 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:56,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:56,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:56,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:56,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:56,464 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:56,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:56,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:56,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,466 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:56,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:56,467 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:56,467 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,467 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:56,468 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:56,469 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@113b1494[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:56,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:56,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,474 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:56,478 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:56,480 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:56,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:56,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:56,481 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:56,481 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:56,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,482 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,483 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,484 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,487 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,487 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:56,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:56,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,488 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:56,488 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:56,588 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:56,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:56,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:56,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:56,592 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:56,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:56,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:56,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,592 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,593 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,593 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,593 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,594 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:56,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:56,595 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,596 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:56,596 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:56,596 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:56,598 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,600 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:56,601 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:56,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6798b9df[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:56,601 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,602 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:56,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:56,602 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:56,602 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,602 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:56,603 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:56,607 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:56,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:56,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:56,607 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:56,607 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:56,608 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,609 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,609 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,610 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,614 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,614 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:56,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:56,616 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:56,616 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:56,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:56,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405816682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,716 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:56,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:56,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:56,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:56,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:56,719 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:56,719 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,719 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:56,719 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:56,720 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,720 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,721 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:56,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:56,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:56,723 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:56,724 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,724 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,725 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,725 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3c16c5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:56,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:56,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,727 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:56,727 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,727 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:56,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:56,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,730 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:56,732 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:56,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:56,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:56,732 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:56,732 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:56,734 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:56,734 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,734 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,734 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,737 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:56,741 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:56,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:56,741 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:56,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,742 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:56,742 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:56,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:56,786 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/64cc6ee5b3dc4af4a0dc3decf13e3f52 2023-04-25 06:55:56,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/79affef4a8fb4a08a2df04fd24191149 2023-04-25 06:55:56,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/014f8cf887184deb833ebad064e6a669 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/014f8cf887184deb833ebad064e6a669 2023-04-25 06:55:56,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/014f8cf887184deb833ebad064e6a669, entries=100, sequenceid=378, filesize=9.5 K 2023-04-25 06:55:56,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/64cc6ee5b3dc4af4a0dc3decf13e3f52 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/64cc6ee5b3dc4af4a0dc3decf13e3f52 2023-04-25 06:55:56,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/64cc6ee5b3dc4af4a0dc3decf13e3f52, entries=100, sequenceid=378, filesize=9.5 K 2023-04-25 06:55:56,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/79affef4a8fb4a08a2df04fd24191149 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/79affef4a8fb4a08a2df04fd24191149 2023-04-25 06:55:56,842 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:56,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:56,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:56,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/79affef4a8fb4a08a2df04fd24191149, entries=100, sequenceid=378, filesize=9.5 K 2023-04-25 06:55:56,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:56,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:56,847 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:56,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:56,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:56,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,847 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,849 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:56,849 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:56,850 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:56,850 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:56,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:56,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:56,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:56,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 75fc43298851a5c4f1060516f9ac39d4 in 898ms, sequenceid=378, compaction requested=true 2023-04-25 06:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:56,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:56,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:56,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:56,853 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:56,853 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:56,853 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:55:56,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 06:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:56,854 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:56,854 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:55:56,854 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,854 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/66f0f6c5ac9547f5b14133c649f898dd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1bd18ff00e104bc58ab27b3f13c50a16, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/014f8cf887184deb833ebad064e6a669] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=34.0 K 2023-04-25 06:55:56,855 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:56,855 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:55:56,855 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,855 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/011d49824b1747b196b09fdec8561280, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8dc9ee34ed7b4459be23dc04fd687bfa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/64cc6ee5b3dc4af4a0dc3decf13e3f52] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=34.0 K 2023-04-25 06:55:56,855 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 66f0f6c5ac9547f5b14133c649f898dd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682405754916 2023-04-25 06:55:56,855 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 011d49824b1747b196b09fdec8561280, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682405754916 2023-04-25 06:55:56,855 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 1bd18ff00e104bc58ab27b3f13c50a16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682405754998 2023-04-25 06:55:56,856 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 8dc9ee34ed7b4459be23dc04fd687bfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682405754998 2023-04-25 06:55:56,856 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 014f8cf887184deb833ebad064e6a669, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682405755637 2023-04-25 06:55:56,856 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 64cc6ee5b3dc4af4a0dc3decf13e3f52, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682405755637 2023-04-25 06:55:56,865 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:56,869 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:56,870 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b6806466ba6d46ed91d8913f405312df 2023-04-25 06:55:56,891 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/70b3b88d318b45fe81dfe6f128b650de as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/70b3b88d318b45fe81dfe6f128b650de 2023-04-25 06:55:56,896 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into 70b3b88d318b45fe81dfe6f128b650de(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:56,896 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:56,896 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=13, startTime=1682405756853; duration=0sec 2023-04-25 06:55:56,897 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:56,897 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:55:56,898 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:55:56,898 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:55:56,898 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:56,898 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b199729201c241a59005aeef8247bc62, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/538b79639c2240ec837cf5b771d8dc8d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/79affef4a8fb4a08a2df04fd24191149] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=34.0 K 2023-04-25 06:55:56,898 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b199729201c241a59005aeef8247bc62, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682405754916 2023-04-25 06:55:56,898 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 538b79639c2240ec837cf5b771d8dc8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682405754998 2023-04-25 06:55:56,899 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 79affef4a8fb4a08a2df04fd24191149, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682405755637 2023-04-25 06:55:56,904 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#444 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:57,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:57,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:57,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405817182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405817182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405817183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405817190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,287 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/6145eadd8a6f489ea61c1e7376fddcaf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/6145eadd8a6f489ea61c1e7376fddcaf 2023-04-25 06:55:57,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405817289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405817289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405817289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,291 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into 6145eadd8a6f489ea61c1e7376fddcaf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:57,291 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:57,291 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=13, startTime=1682405756853; duration=0sec 2023-04-25 06:55:57,291 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:57,298 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/a094f0095cd04890b6b91c147123128d 2023-04-25 06:55:57,305 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/a9142aef74ea4033b8a7896ef3004297 2023-04-25 06:55:57,308 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/b6806466ba6d46ed91d8913f405312df as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b6806466ba6d46ed91d8913f405312df 2023-04-25 06:55:57,312 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b6806466ba6d46ed91d8913f405312df, entries=150, sequenceid=398, filesize=11.9 K 2023-04-25 06:55:57,312 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/a094f0095cd04890b6b91c147123128d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a094f0095cd04890b6b91c147123128d 2023-04-25 06:55:57,313 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/f0348d7fe9264b0fb08fec2a161300d6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f0348d7fe9264b0fb08fec2a161300d6 2023-04-25 06:55:57,315 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a094f0095cd04890b6b91c147123128d, entries=150, sequenceid=398, filesize=11.9 K 2023-04-25 06:55:57,317 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/a9142aef74ea4033b8a7896ef3004297 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a9142aef74ea4033b8a7896ef3004297 2023-04-25 06:55:57,320 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a9142aef74ea4033b8a7896ef3004297, entries=150, sequenceid=398, filesize=11.9 K 2023-04-25 06:55:57,321 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 75fc43298851a5c4f1060516f9ac39d4 in 468ms, sequenceid=398, compaction requested=false 2023-04-25 06:55:57,321 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:57,321 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:57,322 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into f0348d7fe9264b0fb08fec2a161300d6(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:57,322 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:57,322 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=13, startTime=1682405756853; duration=0sec 2023-04-25 06:55:57,322 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,323 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de37b2b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:57,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:57,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,326 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:57,326 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,326 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:57,328 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:57,331 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:57,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:57,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:57,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:57,331 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:57,333 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,333 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,333 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:57,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:57,334 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:57,335 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:57,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:57,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:57,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:57,338 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:57,338 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:57,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,439 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:57,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:57,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:57,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:57,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:57,442 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:57,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:57,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:57,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:57,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:57,444 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:57,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:57,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:57,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:57,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:57,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:57,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:57,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:57,445 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:55:57,446 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:57,446 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:57,446 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:57,446 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:57,446 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:57,446 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:57,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:57,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:57,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405817524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405817525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405817526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405817627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405817632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405817632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405817830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405817836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:57,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405817837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:57,854 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/fe1fb73abc404f8786d304b6c8eae79e 2023-04-25 06:55:58,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405818135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405818142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405818142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56530 deadline: 1682405818204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,271 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/c74dd24c0a0d403387479203c9eeb170 2023-04-25 06:55:58,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56514 deadline: 1682405818410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,416 DEBUG [Thread-2940] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18297 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:55:58,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405818643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405818647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:58,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405818649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,690 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/e66733bf365d4e09b61fb01ba9cad021 2023-04-25 06:55:58,695 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/fe1fb73abc404f8786d304b6c8eae79e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fe1fb73abc404f8786d304b6c8eae79e 2023-04-25 06:55:58,699 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fe1fb73abc404f8786d304b6c8eae79e, entries=150, sequenceid=413, filesize=11.9 K 2023-04-25 06:55:58,700 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/c74dd24c0a0d403387479203c9eeb170 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/c74dd24c0a0d403387479203c9eeb170 2023-04-25 06:55:58,703 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/c74dd24c0a0d403387479203c9eeb170, entries=150, sequenceid=413, filesize=11.9 K 2023-04-25 06:55:58,704 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/e66733bf365d4e09b61fb01ba9cad021 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e66733bf365d4e09b61fb01ba9cad021 2023-04-25 06:55:58,707 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e66733bf365d4e09b61fb01ba9cad021, entries=150, sequenceid=413, filesize=11.9 K 2023-04-25 06:55:58,708 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 75fc43298851a5c4f1060516f9ac39d4 in 1263ms, sequenceid=413, compaction requested=true 2023-04-25 06:55:58,708 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:58,708 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:58,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:58,708 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:58,710 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:58,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:58,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,711 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@643545b7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:58,713 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:58,713 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,715 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:58,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:58,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:58,718 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:58,718 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:58,721 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:58,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:58,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:58,721 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:58,721 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:58,722 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,723 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:58,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,723 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:58,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:58,725 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:58,726 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,729 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,729 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:58,729 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:58,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:58,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:58,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:58,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,730 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:58,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:58,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:58,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:58,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:58,730 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,730 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:58,730 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:58,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:58,830 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:58,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:58,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:58,834 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:58,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:58,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:58,834 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:58,835 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,835 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:58,836 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:58,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:58,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:58,837 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:58,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:58,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:58,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:58,838 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:58,838 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:58,838 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:55:58,839 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:58,839 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:58,839 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:58,839 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:58,839 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:58,839 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:58,847 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/6f17472182c341a193c430131daf3487 2023-04-25 06:55:58,857 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/cf7a95fc896141139154eac64cf966b6 2023-04-25 06:55:59,269 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/e7e7f6c533c34039b7a5d33806dd58e4 2023-04-25 06:55:59,273 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/6f17472182c341a193c430131daf3487 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/6f17472182c341a193c430131daf3487 2023-04-25 06:55:59,277 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/6f17472182c341a193c430131daf3487, entries=150, sequenceid=433, filesize=11.9 K 2023-04-25 06:55:59,277 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/cf7a95fc896141139154eac64cf966b6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cf7a95fc896141139154eac64cf966b6 2023-04-25 06:55:59,281 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cf7a95fc896141139154eac64cf966b6, entries=150, sequenceid=433, filesize=11.9 K 2023-04-25 06:55:59,282 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/e7e7f6c533c34039b7a5d33806dd58e4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e7e7f6c533c34039b7a5d33806dd58e4 2023-04-25 06:55:59,286 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e7e7f6c533c34039b7a5d33806dd58e4, entries=150, sequenceid=433, filesize=11.9 K 2023-04-25 06:55:59,287 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 75fc43298851a5c4f1060516f9ac39d4 in 449ms, sequenceid=433, compaction requested=true 2023-04-25 06:55:59,287 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:59,287 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:59,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:59,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:59,287 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:59,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46414eaf[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:59,292 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:59,292 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,293 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,293 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:59,293 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:59,293 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:59,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,293 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,294 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:59,297 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:59,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:59,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:59,297 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:59,297 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:59,300 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,300 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,300 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,301 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,304 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,306 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:59,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,306 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,307 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,307 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:59,307 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:59,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,407 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:59,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:59,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:59,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:59,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:59,411 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:59,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:59,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:59,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,411 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,415 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,415 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,415 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,415 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:59,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:59,417 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,417 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:59,417 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:59,417 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:59,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:59,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:59,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59d4e29c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:59,423 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:59,425 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:59,425 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,425 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,425 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:59,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:59,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:59,426 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:59,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:59,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:59,426 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:59,426 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:59,427 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,427 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,427 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,428 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,431 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,434 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:59,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,435 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:59,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:59,435 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:59,435 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:59,535 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:59,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:59,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:59,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:59,539 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:59,539 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,539 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:59,539 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:59,539 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,539 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,541 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,541 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:59,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:59,543 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,543 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:59,543 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:59,543 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:59,545 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f58e0ae[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:59,547 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:59,547 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,548 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,548 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,548 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:59,551 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:59,551 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,551 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:59,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:59,554 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:59,554 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:59,554 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:59,554 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:59,554 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:59,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,555 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:59,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,560 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:59,560 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:59,560 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:59,660 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:59,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:59,664 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:59,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:59,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:59,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,665 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:59,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:55:59,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:59,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:59,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:59,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:59,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:59,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:59,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:59,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:59,668 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,668 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:59,668 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:59,668 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,671 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/cca6385b8450499cb2bbc6af139d8434 2023-04-25 06:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@210e080f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:59,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:59,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,676 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:59,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,678 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:59,683 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:59,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:59,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:59,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:59,683 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:59,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/47bde24589394f5381d567d31da3a867 2023-04-25 06:55:59,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/9a9d9fcbaaca48e4a4b63c4ed714751d 2023-04-25 06:55:59,694 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,694 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,694 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,694 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:55:59,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:59,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:59,695 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,695 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,695 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:59,696 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:59,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/cca6385b8450499cb2bbc6af139d8434 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cca6385b8450499cb2bbc6af139d8434 2023-04-25 06:55:59,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cca6385b8450499cb2bbc6af139d8434, entries=250, sequenceid=445, filesize=16.7 K 2023-04-25 06:55:59,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/47bde24589394f5381d567d31da3a867 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47bde24589394f5381d567d31da3a867 2023-04-25 06:55:59,706 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47bde24589394f5381d567d31da3a867, entries=150, sequenceid=445, filesize=11.9 K 2023-04-25 06:55:59,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/9a9d9fcbaaca48e4a4b63c4ed714751d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9a9d9fcbaaca48e4a4b63c4ed714751d 2023-04-25 06:55:59,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9a9d9fcbaaca48e4a4b63c4ed714751d, entries=150, sequenceid=445, filesize=11.9 K 2023-04-25 06:55:59,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=73.80 KB/75570 for 75fc43298851a5c4f1060516f9ac39d4 in 44ms, sequenceid=445, compaction requested=true 2023-04-25 06:55:59,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:59,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:59,710 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:55:59,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:59,710 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:55:59,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:59,712 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66594 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:55:59,712 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:55:59,712 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,712 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/70b3b88d318b45fe81dfe6f128b650de, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b6806466ba6d46ed91d8913f405312df, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fe1fb73abc404f8786d304b6c8eae79e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/6f17472182c341a193c430131daf3487, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cca6385b8450499cb2bbc6af139d8434] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=65.0 K 2023-04-25 06:55:59,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:55:59,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:55:59,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:55:59,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:59,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:55:59,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:59,713 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:55:59,713 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 70b3b88d318b45fe81dfe6f128b650de, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682405755009 2023-04-25 06:55:59,713 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:55:59,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:55:59,714 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:55:59,714 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/6145eadd8a6f489ea61c1e7376fddcaf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a094f0095cd04890b6b91c147123128d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/c74dd24c0a0d403387479203c9eeb170, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cf7a95fc896141139154eac64cf966b6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47bde24589394f5381d567d31da3a867] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=60.3 K 2023-04-25 06:55:59,714 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b6806466ba6d46ed91d8913f405312df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1682405756045 2023-04-25 06:55:59,714 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6145eadd8a6f489ea61c1e7376fddcaf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682405755009 2023-04-25 06:55:59,714 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting fe1fb73abc404f8786d304b6c8eae79e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405757169 2023-04-25 06:55:59,714 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a094f0095cd04890b6b91c147123128d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1682405756045 2023-04-25 06:55:59,715 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6f17472182c341a193c430131daf3487, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682405757522 2023-04-25 06:55:59,715 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c74dd24c0a0d403387479203c9eeb170, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405757169 2023-04-25 06:55:59,715 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting cca6385b8450499cb2bbc6af139d8434, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405759653 2023-04-25 06:55:59,715 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting cf7a95fc896141139154eac64cf966b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682405757522 2023-04-25 06:55:59,716 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 47bde24589394f5381d567d31da3a867, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405759653 2023-04-25 06:55:59,727 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#457 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:59,727 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#456 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:59,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/fa39d30f4db24332a4e1a0ffc9c4327f 2023-04-25 06:55:59,739 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/4bfb408e63f64d6dacca9dd0e7bc95d7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bfb408e63f64d6dacca9dd0e7bc95d7 2023-04-25 06:55:59,744 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into 4bfb408e63f64d6dacca9dd0e7bc95d7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:59,744 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:59,744 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=11, startTime=1682405759710; duration=0sec 2023-04-25 06:55:59,745 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:55:59,745 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:55:59,746 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:55:59,746 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:55:59,747 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,747 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f0348d7fe9264b0fb08fec2a161300d6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a9142aef74ea4033b8a7896ef3004297, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e66733bf365d4e09b61fb01ba9cad021, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e7e7f6c533c34039b7a5d33806dd58e4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9a9d9fcbaaca48e4a4b63c4ed714751d] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=60.3 K 2023-04-25 06:55:59,747 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting f0348d7fe9264b0fb08fec2a161300d6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682405755009 2023-04-25 06:55:59,747 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a9142aef74ea4033b8a7896ef3004297, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1682405756045 2023-04-25 06:55:59,748 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting e66733bf365d4e09b61fb01ba9cad021, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682405757169 2023-04-25 06:55:59,748 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting e7e7f6c533c34039b7a5d33806dd58e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682405757522 2023-04-25 06:55:59,748 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9a9d9fcbaaca48e4a4b63c4ed714751d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405759653 2023-04-25 06:55:59,750 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/814abc0aeeb5439fa8a6872c2e845d6e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/814abc0aeeb5439fa8a6872c2e845d6e 2023-04-25 06:55:59,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/42da1ff492d84903b3e07e9d8c04ca09 2023-04-25 06:55:59,755 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into 814abc0aeeb5439fa8a6872c2e845d6e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:55:59,755 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:55:59,755 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=11, startTime=1682405759710; duration=0sec 2023-04-25 06:55:59,755 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:55:59,757 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#460 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:55:59,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:59,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405819748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:59,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405819759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:59,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405819761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,796 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:59,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:59,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:59,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:59,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:59,799 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:59,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:59,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:59,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,800 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:59,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:59,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:59,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:59,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:59,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:59,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:59,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:59,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df60eb7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:59,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,810 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:59,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:59,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:59,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:59,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:59,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:59,814 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:59,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:59,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:59,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:59,814 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:59,816 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,816 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,816 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,817 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:59,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:59,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,823 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:59,823 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:59,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:59,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405819862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:59,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405819862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:55:59,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405819869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,923 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:55:59,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:55:59,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:55:59,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:55:59,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:55:59,926 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:55:59,926 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:55:59,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:55:59,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,929 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:55:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:55:59,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:55:59,931 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,931 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:55:59,931 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:55:59,931 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:55:59,933 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,933 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,933 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,933 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:55:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:55:59,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1348d197[Count = 0] remaining members to acquire global barrier 2023-04-25 06:55:59,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:55:59,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,936 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:55:59,936 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,936 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:55:59,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:55:59,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,937 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:55:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:55:59,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:55:59,940 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:55:59,940 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:55:59,940 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:55:59,940 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:55:59,940 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:55:59,942 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,942 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:55:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:55:59,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,942 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:55:59,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:55:59,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:55:59,945 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:55:59,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:55:59,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:55:59,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:55:59,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:55:59,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:55:59,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:55:59,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:55:59,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:55:59,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:55:59,947 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:55:59,947 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:55:59,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:00,047 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:00,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:00,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:00,051 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:00,051 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:00,051 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:00,051 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:00,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,052 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:00,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:00,053 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:00,053 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,053 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:00,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,054 DEBUG [Thread-2959] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36d2d826 to 127.0.0.1:60416 2023-04-25 06:56:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:00,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:00,054 DEBUG [Thread-2959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:00,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:00,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:00,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. started... 2023-04-25 06:56:00,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. as already flushing 2023-04-25 06:56:00,057 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:00,058 DEBUG [Thread-2955] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24a917df to 127.0.0.1:60416 2023-04-25 06:56:00,058 DEBUG [Thread-2955] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,059 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:00,061 DEBUG [Thread-2951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x028a3a77 to 127.0.0.1:60416 2023-04-25 06:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:00,061 DEBUG [Thread-2951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,061 DEBUG [Thread-2953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cefb012 to 127.0.0.1:60416 2023-04-25 06:56:00,061 DEBUG [Thread-2953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:00,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:00,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbd0ec0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:00,063 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:00,063 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,064 DEBUG [Thread-2957] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78fa3dee to 127.0.0.1:60416 2023-04-25 06:56:00,064 DEBUG [Thread-2957] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,065 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:00,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:00,067 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,067 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:00,067 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:00,067 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56540 deadline: 1682405820068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56488 deadline: 1682405820068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:00,069 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:00,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:00,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:00,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:00,069 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:00,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:00,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,070 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:00,071 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:00,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:00,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:00,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:00,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:00,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:00,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:00,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:00,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:00,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:00,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56528 deadline: 1682405820074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:00,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:00,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:00,075 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:00,075 DEBUG [Thread-2950] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:00,075 DEBUG [Thread-2950] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:00,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:00,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/51aed01c1a914002ac229f715eb045ed 2023-04-25 06:56:00,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/fa39d30f4db24332a4e1a0ffc9c4327f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fa39d30f4db24332a4e1a0ffc9c4327f 2023-04-25 06:56:00,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fa39d30f4db24332a4e1a0ffc9c4327f, entries=250, sequenceid=461, filesize=16.7 K 2023-04-25 06:56:00,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/42da1ff492d84903b3e07e9d8c04ca09 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/42da1ff492d84903b3e07e9d8c04ca09 2023-04-25 06:56:00,172 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/75595679523b4ad1a341696aa0ab1fca as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/75595679523b4ad1a341696aa0ab1fca 2023-04-25 06:56:00,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/42da1ff492d84903b3e07e9d8c04ca09, entries=150, sequenceid=461, filesize=11.9 K 2023-04-25 06:56:00,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/51aed01c1a914002ac229f715eb045ed as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/51aed01c1a914002ac229f715eb045ed 2023-04-25 06:56:00,176 DEBUG [Thread-2950] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:00,176 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into 75595679523b4ad1a341696aa0ab1fca(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:00,176 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:56:00,176 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=11, startTime=1682405759710; duration=0sec 2023-04-25 06:56:00,176 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:00,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/51aed01c1a914002ac229f715eb045ed, entries=150, sequenceid=461, filesize=11.9 K 2023-04-25 06:56:00,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:00,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 75fc43298851a5c4f1060516f9ac39d4 in 465ms, sequenceid=461, compaction requested=false 2023-04-25 06:56:00,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:56:00,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:56:00,223 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:56:00,224 DEBUG [Thread-2942] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x357cb237 to 127.0.0.1:60416 2023-04-25 06:56:00,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:56:00,224 DEBUG [Thread-2942] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:00,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:56:00,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:00,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:56:00,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:00,371 DEBUG [Thread-2944] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4539ed0c to 127.0.0.1:60416 2023-04-25 06:56:00,371 DEBUG [Thread-2946] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32cd672d to 127.0.0.1:60416 2023-04-25 06:56:00,371 DEBUG [Thread-2944] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,371 DEBUG [Thread-2946] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,377 DEBUG [Thread-2948] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09b8ff30 to 127.0.0.1:60416 2023-04-25 06:56:00,377 DEBUG [Thread-2948] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:00,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/87b8bd2e4e5a4901ac8cfb6365a3db01 2023-04-25 06:56:01,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/2c88e290b3444ec98e8473ea564d305c 2023-04-25 06:56:01,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/7c0d44eb6ca5467ea7381616623b2fa8 2023-04-25 06:56:01,452 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/87b8bd2e4e5a4901ac8cfb6365a3db01 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/87b8bd2e4e5a4901ac8cfb6365a3db01 2023-04-25 06:56:01,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/87b8bd2e4e5a4901ac8cfb6365a3db01, entries=150, sequenceid=481, filesize=11.9 K 2023-04-25 06:56:01,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/2c88e290b3444ec98e8473ea564d305c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/2c88e290b3444ec98e8473ea564d305c 2023-04-25 06:56:01,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/2c88e290b3444ec98e8473ea564d305c, entries=150, sequenceid=481, filesize=11.9 K 2023-04-25 06:56:01,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/7c0d44eb6ca5467ea7381616623b2fa8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7c0d44eb6ca5467ea7381616623b2fa8 2023-04-25 06:56:01,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7c0d44eb6ca5467ea7381616623b2fa8, entries=150, sequenceid=481, filesize=11.9 K 2023-04-25 06:56:01,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=20.13 KB/20610 for 75fc43298851a5c4f1060516f9ac39d4 in 1239ms, sequenceid=481, compaction requested=true 2023-04-25 06:56:01,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:56:01,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:01,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:01,462 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:01,462 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:01,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:01,463 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42392 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:01,463 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:01,463 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/A is initiating minor compaction (all files) 2023-04-25 06:56:01,463 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/B is initiating minor compaction (all files) 2023-04-25 06:56:01,463 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/A in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:01,463 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/B in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:01,463 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bfb408e63f64d6dacca9dd0e7bc95d7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fa39d30f4db24332a4e1a0ffc9c4327f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/87b8bd2e4e5a4901ac8cfb6365a3db01] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=41.4 K 2023-04-25 06:56:01,463 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/814abc0aeeb5439fa8a6872c2e845d6e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/42da1ff492d84903b3e07e9d8c04ca09, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/2c88e290b3444ec98e8473ea564d305c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=36.6 K 2023-04-25 06:56:01,464 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 4bfb408e63f64d6dacca9dd0e7bc95d7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405759653 2023-04-25 06:56:01,464 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 814abc0aeeb5439fa8a6872c2e845d6e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405759653 2023-04-25 06:56:01,464 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting fa39d30f4db24332a4e1a0ffc9c4327f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1682405759697 2023-04-25 06:56:01,464 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 42da1ff492d84903b3e07e9d8c04ca09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1682405759697 2023-04-25 06:56:01,464 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 87b8bd2e4e5a4901ac8cfb6365a3db01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1682405759746 2023-04-25 06:56:01,464 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2c88e290b3444ec98e8473ea564d305c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1682405759746 2023-04-25 06:56:01,470 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#B#compaction#464 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:01,470 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#A#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:01,877 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/98cbaad8dcdb46caaff5813d4cfdad2a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/98cbaad8dcdb46caaff5813d4cfdad2a 2023-04-25 06:56:01,877 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/cdcf8af2798145e69b8ae9a48bcf0aa8 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cdcf8af2798145e69b8ae9a48bcf0aa8 2023-04-25 06:56:01,881 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/B of 75fc43298851a5c4f1060516f9ac39d4 into cdcf8af2798145e69b8ae9a48bcf0aa8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:01,881 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/A of 75fc43298851a5c4f1060516f9ac39d4 into 98cbaad8dcdb46caaff5813d4cfdad2a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:01,881 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:56:01,881 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:56:01,881 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/B, priority=13, startTime=1682405761462; duration=0sec 2023-04-25 06:56:01,881 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/A, priority=13, startTime=1682405761462; duration=0sec 2023-04-25 06:56:01,881 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:01,881 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:01,881 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:01,882 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:01,882 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): 75fc43298851a5c4f1060516f9ac39d4/C is initiating minor compaction (all files) 2023-04-25 06:56:01,882 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 75fc43298851a5c4f1060516f9ac39d4/C in TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:01,882 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/75595679523b4ad1a341696aa0ab1fca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/51aed01c1a914002ac229f715eb045ed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7c0d44eb6ca5467ea7381616623b2fa8] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp, totalSize=36.6 K 2023-04-25 06:56:01,882 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 75595679523b4ad1a341696aa0ab1fca, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682405759653 2023-04-25 06:56:01,882 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 51aed01c1a914002ac229f715eb045ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1682405759697 2023-04-25 06:56:01,882 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7c0d44eb6ca5467ea7381616623b2fa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1682405759746 2023-04-25 06:56:01,887 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): 75fc43298851a5c4f1060516f9ac39d4#C#compaction#466 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:02,295 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/f3617f2fa17f4a008c5155b0634ce6a5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f3617f2fa17f4a008c5155b0634ce6a5 2023-04-25 06:56:02,298 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 75fc43298851a5c4f1060516f9ac39d4/C of 75fc43298851a5c4f1060516f9ac39d4 into f3617f2fa17f4a008c5155b0634ce6a5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:02,298 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:56:02,298 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4., storeName=75fc43298851a5c4f1060516f9ac39d4/C, priority=13, startTime=1682405761462; duration=0sec 2023-04-25 06:56:02,298 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:08,436 DEBUG [Thread-2940] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fb41c50 to 127.0.0.1:60416 2023-04-25 06:56:08,436 DEBUG [Thread-2940] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:08,436 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 9 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 95 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1746 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5234 rows 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1750 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5246 rows 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1742 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5225 rows 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1738 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5212 rows 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(402): scanned 1734 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(403): verified 5194 rows 2023-04-25 06:56:08,437 INFO [Listener at localhost/41663] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 06:56:08,437 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c8fbc35 to 127.0.0.1:60416 2023-04-25 06:56:08,437 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:08,441 INFO [Listener at localhost/41663] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 06:56:08,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 06:56:08,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:08,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 06:56:08,445 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405768445"}]},"ts":"1682405768445"} 2023-04-25 06:56:08,446 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 06:56:08,447 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 06:56:08,448 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75fc43298851a5c4f1060516f9ac39d4, UNASSIGN}] 2023-04-25 06:56:08,449 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=75fc43298851a5c4f1060516f9ac39d4, UNASSIGN 2023-04-25 06:56:08,450 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=75fc43298851a5c4f1060516f9ac39d4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:08,450 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405768449"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405768449"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405768449"}]},"ts":"1682405768449"} 2023-04-25 06:56:08,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:56:08,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 06:56:08,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:56:08,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 75fc43298851a5c4f1060516f9ac39d4, disabling compactions & flushes 2023-04-25 06:56:08,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:08,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:08,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. after waiting 0 ms 2023-04-25 06:56:08,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:08,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 75fc43298851a5c4f1060516f9ac39d4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 06:56:08,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=A 2023-04-25 06:56:08,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:08,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=B 2023-04-25 06:56:08,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:08,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 75fc43298851a5c4f1060516f9ac39d4, store=C 2023-04-25 06:56:08,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:08,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 06:56:09,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/37a7b9e7f909499bab2383ed59b9faed 2023-04-25 06:56:09,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 06:56:09,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/ab369359f1994c98bec1f4f341857968 2023-04-25 06:56:09,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 06:56:09,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/036fd67146dc4d78bc5a32a43ce58f5d 2023-04-25 06:56:09,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/A/37a7b9e7f909499bab2383ed59b9faed as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37a7b9e7f909499bab2383ed59b9faed 2023-04-25 06:56:09,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37a7b9e7f909499bab2383ed59b9faed, entries=150, sequenceid=491, filesize=11.9 K 2023-04-25 06:56:09,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/B/ab369359f1994c98bec1f4f341857968 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ab369359f1994c98bec1f4f341857968 2023-04-25 06:56:09,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ab369359f1994c98bec1f4f341857968, entries=150, sequenceid=491, filesize=11.9 K 2023-04-25 06:56:09,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/.tmp/C/036fd67146dc4d78bc5a32a43ce58f5d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/036fd67146dc4d78bc5a32a43ce58f5d 2023-04-25 06:56:09,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/036fd67146dc4d78bc5a32a43ce58f5d, entries=150, sequenceid=491, filesize=11.9 K 2023-04-25 06:56:09,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 75fc43298851a5c4f1060516f9ac39d4 in 1240ms, sequenceid=491, compaction requested=false 2023-04-25 06:56:09,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cfc945f3e1d949ea9f7fd7fbddafaf96, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/44f94e5eaef44081afdec9e25e075d9b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bb5d726da8e4c649dcc0edc65630ee1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bed1d94a8730432fa2d22dec66c40c27, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1b589d13d996451a80c8cdf969c9179b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/34ba6ba00a994afe924b11ebc119881c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/59bb8199df4a47038571d303d95dd97d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b093d647059a4a9098413fcfcff0bfe2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/58e638452ff34950b57944b72d478b64, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37ca9b96bf3142baae15af0009438659, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/28270bef748a4eb5af0918d77e632073, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/340ca07384e745ea9af24801de2b85d6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/c0d0e2d4977d4e85ab4375ad74617d03, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/d97868e8661f41d3b0fb475a97189aee, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cb70995b592e4ddc9974c0aa892eeec9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/5d5e05f7a44a40d1b3e352250ebf668b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b9eda3d2aae246bab3b79e92ebb8d53d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b5ad1b35e54c4e3f80b230558af9aab1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/612d71e9c9604d9eb4bd9e037260c361, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/2558aab9aff744518fe55ffe9be4877f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/66f0f6c5ac9547f5b14133c649f898dd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1bd18ff00e104bc58ab27b3f13c50a16, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/70b3b88d318b45fe81dfe6f128b650de, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/014f8cf887184deb833ebad064e6a669, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b6806466ba6d46ed91d8913f405312df, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fe1fb73abc404f8786d304b6c8eae79e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/6f17472182c341a193c430131daf3487, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cca6385b8450499cb2bbc6af139d8434, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bfb408e63f64d6dacca9dd0e7bc95d7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fa39d30f4db24332a4e1a0ffc9c4327f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/87b8bd2e4e5a4901ac8cfb6365a3db01] to archive 2023-04-25 06:56:09,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:56:09,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cfc945f3e1d949ea9f7fd7fbddafaf96 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cfc945f3e1d949ea9f7fd7fbddafaf96 2023-04-25 06:56:09,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/44f94e5eaef44081afdec9e25e075d9b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/44f94e5eaef44081afdec9e25e075d9b 2023-04-25 06:56:09,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bb5d726da8e4c649dcc0edc65630ee1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bb5d726da8e4c649dcc0edc65630ee1 2023-04-25 06:56:09,870 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bed1d94a8730432fa2d22dec66c40c27 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/bed1d94a8730432fa2d22dec66c40c27 2023-04-25 06:56:09,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1b589d13d996451a80c8cdf969c9179b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1b589d13d996451a80c8cdf969c9179b 2023-04-25 06:56:09,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/34ba6ba00a994afe924b11ebc119881c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/34ba6ba00a994afe924b11ebc119881c 2023-04-25 06:56:09,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/59bb8199df4a47038571d303d95dd97d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/59bb8199df4a47038571d303d95dd97d 2023-04-25 06:56:09,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b093d647059a4a9098413fcfcff0bfe2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b093d647059a4a9098413fcfcff0bfe2 2023-04-25 06:56:09,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/58e638452ff34950b57944b72d478b64 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/58e638452ff34950b57944b72d478b64 2023-04-25 06:56:09,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37ca9b96bf3142baae15af0009438659 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37ca9b96bf3142baae15af0009438659 2023-04-25 06:56:09,875 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/28270bef748a4eb5af0918d77e632073 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/28270bef748a4eb5af0918d77e632073 2023-04-25 06:56:09,876 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/340ca07384e745ea9af24801de2b85d6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/340ca07384e745ea9af24801de2b85d6 2023-04-25 06:56:09,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/c0d0e2d4977d4e85ab4375ad74617d03 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/c0d0e2d4977d4e85ab4375ad74617d03 2023-04-25 06:56:09,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/d97868e8661f41d3b0fb475a97189aee to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/d97868e8661f41d3b0fb475a97189aee 2023-04-25 06:56:09,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cb70995b592e4ddc9974c0aa892eeec9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cb70995b592e4ddc9974c0aa892eeec9 2023-04-25 06:56:09,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/5d5e05f7a44a40d1b3e352250ebf668b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/5d5e05f7a44a40d1b3e352250ebf668b 2023-04-25 06:56:09,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b9eda3d2aae246bab3b79e92ebb8d53d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b9eda3d2aae246bab3b79e92ebb8d53d 2023-04-25 06:56:09,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b5ad1b35e54c4e3f80b230558af9aab1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b5ad1b35e54c4e3f80b230558af9aab1 2023-04-25 06:56:09,881 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/612d71e9c9604d9eb4bd9e037260c361 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/612d71e9c9604d9eb4bd9e037260c361 2023-04-25 06:56:09,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/2558aab9aff744518fe55ffe9be4877f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/2558aab9aff744518fe55ffe9be4877f 2023-04-25 06:56:09,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/66f0f6c5ac9547f5b14133c649f898dd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/66f0f6c5ac9547f5b14133c649f898dd 2023-04-25 06:56:09,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1bd18ff00e104bc58ab27b3f13c50a16 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/1bd18ff00e104bc58ab27b3f13c50a16 2023-04-25 06:56:09,884 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/70b3b88d318b45fe81dfe6f128b650de to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/70b3b88d318b45fe81dfe6f128b650de 2023-04-25 06:56:09,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/014f8cf887184deb833ebad064e6a669 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/014f8cf887184deb833ebad064e6a669 2023-04-25 06:56:09,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b6806466ba6d46ed91d8913f405312df to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/b6806466ba6d46ed91d8913f405312df 2023-04-25 06:56:09,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fe1fb73abc404f8786d304b6c8eae79e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fe1fb73abc404f8786d304b6c8eae79e 2023-04-25 06:56:09,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/6f17472182c341a193c430131daf3487 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/6f17472182c341a193c430131daf3487 2023-04-25 06:56:09,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cca6385b8450499cb2bbc6af139d8434 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/cca6385b8450499cb2bbc6af139d8434 2023-04-25 06:56:09,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bfb408e63f64d6dacca9dd0e7bc95d7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/4bfb408e63f64d6dacca9dd0e7bc95d7 2023-04-25 06:56:09,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fa39d30f4db24332a4e1a0ffc9c4327f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/fa39d30f4db24332a4e1a0ffc9c4327f 2023-04-25 06:56:09,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/87b8bd2e4e5a4901ac8cfb6365a3db01 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/87b8bd2e4e5a4901ac8cfb6365a3db01 2023-04-25 06:56:09,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ee49a5d4b3b5463388cdbc6c5aa86ecc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4b09c574f203497a9b7ad554f72108b5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ff560ff0543142a4ab1f3783071692ca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/10d57afc62bd4745ac45d31b9ec67f06, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cabec64139514e8481635a8ba9fdab18, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/da5b4d974f81439491b996dc91363ea8, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8896ba374993461d9f32e70309844450, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/87f9bc542e1448e59bb04992b97a6fa6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/5900f70c58e14022afca58bdf816eb9c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/fe9297f7e4a646ad92f3c188a883242b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9f2a07b8b43847b68f74c15d50d6d200, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4494c6d2b0a5418cac653f643689c778, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/340394ee2e6647c68ba49adebffb68f5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9bb1afb1ed644477a4a9d6bb957560fa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47e13b17b72d4eb69e7a489e35a3ccdf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/f1126ad0f8bc4a97a2bc2f7bfa5ac3b3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/05e8c0e3a24b40409775a19b46b77d32, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/623d117b10024b3aa74c61e1d9f0a49d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/68861ec9f110418488007c111d0765f7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/011d49824b1747b196b09fdec8561280, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/0ca7decfedab42cdbbd10c11565f4959, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8dc9ee34ed7b4459be23dc04fd687bfa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/6145eadd8a6f489ea61c1e7376fddcaf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/64cc6ee5b3dc4af4a0dc3decf13e3f52, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a094f0095cd04890b6b91c147123128d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/c74dd24c0a0d403387479203c9eeb170, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cf7a95fc896141139154eac64cf966b6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/814abc0aeeb5439fa8a6872c2e845d6e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47bde24589394f5381d567d31da3a867, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/42da1ff492d84903b3e07e9d8c04ca09, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/2c88e290b3444ec98e8473ea564d305c] to archive 2023-04-25 06:56:09,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:56:09,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ee49a5d4b3b5463388cdbc6c5aa86ecc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ee49a5d4b3b5463388cdbc6c5aa86ecc 2023-04-25 06:56:09,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4b09c574f203497a9b7ad554f72108b5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4b09c574f203497a9b7ad554f72108b5 2023-04-25 06:56:09,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ff560ff0543142a4ab1f3783071692ca to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ff560ff0543142a4ab1f3783071692ca 2023-04-25 06:56:09,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/10d57afc62bd4745ac45d31b9ec67f06 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/10d57afc62bd4745ac45d31b9ec67f06 2023-04-25 06:56:09,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cabec64139514e8481635a8ba9fdab18 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cabec64139514e8481635a8ba9fdab18 2023-04-25 06:56:09,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/da5b4d974f81439491b996dc91363ea8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/da5b4d974f81439491b996dc91363ea8 2023-04-25 06:56:09,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8896ba374993461d9f32e70309844450 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8896ba374993461d9f32e70309844450 2023-04-25 06:56:09,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/87f9bc542e1448e59bb04992b97a6fa6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/87f9bc542e1448e59bb04992b97a6fa6 2023-04-25 06:56:09,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/5900f70c58e14022afca58bdf816eb9c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/5900f70c58e14022afca58bdf816eb9c 2023-04-25 06:56:09,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/fe9297f7e4a646ad92f3c188a883242b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/fe9297f7e4a646ad92f3c188a883242b 2023-04-25 06:56:09,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9f2a07b8b43847b68f74c15d50d6d200 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9f2a07b8b43847b68f74c15d50d6d200 2023-04-25 06:56:09,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4494c6d2b0a5418cac653f643689c778 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/4494c6d2b0a5418cac653f643689c778 2023-04-25 06:56:09,905 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/340394ee2e6647c68ba49adebffb68f5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/340394ee2e6647c68ba49adebffb68f5 2023-04-25 06:56:09,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9bb1afb1ed644477a4a9d6bb957560fa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/9bb1afb1ed644477a4a9d6bb957560fa 2023-04-25 06:56:09,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47e13b17b72d4eb69e7a489e35a3ccdf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47e13b17b72d4eb69e7a489e35a3ccdf 2023-04-25 06:56:09,907 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/f1126ad0f8bc4a97a2bc2f7bfa5ac3b3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/f1126ad0f8bc4a97a2bc2f7bfa5ac3b3 2023-04-25 06:56:09,908 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/05e8c0e3a24b40409775a19b46b77d32 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/05e8c0e3a24b40409775a19b46b77d32 2023-04-25 06:56:09,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/623d117b10024b3aa74c61e1d9f0a49d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/623d117b10024b3aa74c61e1d9f0a49d 2023-04-25 06:56:09,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/68861ec9f110418488007c111d0765f7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/68861ec9f110418488007c111d0765f7 2023-04-25 06:56:09,910 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/011d49824b1747b196b09fdec8561280 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/011d49824b1747b196b09fdec8561280 2023-04-25 06:56:09,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/0ca7decfedab42cdbbd10c11565f4959 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/0ca7decfedab42cdbbd10c11565f4959 2023-04-25 06:56:09,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8dc9ee34ed7b4459be23dc04fd687bfa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/8dc9ee34ed7b4459be23dc04fd687bfa 2023-04-25 06:56:09,912 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/6145eadd8a6f489ea61c1e7376fddcaf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/6145eadd8a6f489ea61c1e7376fddcaf 2023-04-25 06:56:09,913 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/64cc6ee5b3dc4af4a0dc3decf13e3f52 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/64cc6ee5b3dc4af4a0dc3decf13e3f52 2023-04-25 06:56:09,914 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a094f0095cd04890b6b91c147123128d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/a094f0095cd04890b6b91c147123128d 2023-04-25 06:56:09,914 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/c74dd24c0a0d403387479203c9eeb170 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/c74dd24c0a0d403387479203c9eeb170 2023-04-25 06:56:09,915 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cf7a95fc896141139154eac64cf966b6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cf7a95fc896141139154eac64cf966b6 2023-04-25 06:56:09,916 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/814abc0aeeb5439fa8a6872c2e845d6e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/814abc0aeeb5439fa8a6872c2e845d6e 2023-04-25 06:56:09,917 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47bde24589394f5381d567d31da3a867 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/47bde24589394f5381d567d31da3a867 2023-04-25 06:56:09,918 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/42da1ff492d84903b3e07e9d8c04ca09 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/42da1ff492d84903b3e07e9d8c04ca09 2023-04-25 06:56:09,918 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/2c88e290b3444ec98e8473ea564d305c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/2c88e290b3444ec98e8473ea564d305c 2023-04-25 06:56:09,921 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c3ad46da3d7041c8ae97824615627cba, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c8f90d8c79403685e91f7c3cc522ed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/92a81ce74b8a4cff991f243d5ad25b17, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7ad2169899ee486cb7749b8caf32bf74, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/82a01267182549cd8bd4ed5c0795317b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c2b81b1000499985946131bc039043, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/395450db8b764637897663a4c7583b4a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b1bb3f253ef148e188b1657cd1fadc5c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d899d98eeacc453d8decd9e98110d8e7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/02feb1d17f9e4d728ef27c32ada9c467, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a5891719067d4954be47149921ae4512, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9764ecf3a16d40c1b8ef41ed83841abb, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7b771d3cb37d49e79ed35170e66edc2b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/14817c7a27c94c22b5955768d2333d68, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/4e89ec1ca9b7480dbfec9454d5882bad, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/349a0471e5574d6484d1854a0d9371f1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/546ae7f060e74cc98c94270b7128d9c7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/122d347388cb48c1af8d69e24c1cc362, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/ff6a745658174b229de2605d3b40aadc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b199729201c241a59005aeef8247bc62, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/011848a908a840cbbbfe7da39cb0a9f5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/538b79639c2240ec837cf5b771d8dc8d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f0348d7fe9264b0fb08fec2a161300d6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/79affef4a8fb4a08a2df04fd24191149, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a9142aef74ea4033b8a7896ef3004297, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e66733bf365d4e09b61fb01ba9cad021, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e7e7f6c533c34039b7a5d33806dd58e4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/75595679523b4ad1a341696aa0ab1fca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9a9d9fcbaaca48e4a4b63c4ed714751d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/51aed01c1a914002ac229f715eb045ed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7c0d44eb6ca5467ea7381616623b2fa8] to archive 2023-04-25 06:56:09,922 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:56:09,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c3ad46da3d7041c8ae97824615627cba to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/c3ad46da3d7041c8ae97824615627cba 2023-04-25 06:56:09,925 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c8f90d8c79403685e91f7c3cc522ed to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c8f90d8c79403685e91f7c3cc522ed 2023-04-25 06:56:09,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/92a81ce74b8a4cff991f243d5ad25b17 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/92a81ce74b8a4cff991f243d5ad25b17 2023-04-25 06:56:09,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7ad2169899ee486cb7749b8caf32bf74 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7ad2169899ee486cb7749b8caf32bf74 2023-04-25 06:56:09,928 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/82a01267182549cd8bd4ed5c0795317b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/82a01267182549cd8bd4ed5c0795317b 2023-04-25 06:56:09,929 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c2b81b1000499985946131bc039043 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/36c2b81b1000499985946131bc039043 2023-04-25 06:56:09,931 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/395450db8b764637897663a4c7583b4a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/395450db8b764637897663a4c7583b4a 2023-04-25 06:56:09,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b1bb3f253ef148e188b1657cd1fadc5c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b1bb3f253ef148e188b1657cd1fadc5c 2023-04-25 06:56:09,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d899d98eeacc453d8decd9e98110d8e7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/d899d98eeacc453d8decd9e98110d8e7 2023-04-25 06:56:09,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/02feb1d17f9e4d728ef27c32ada9c467 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/02feb1d17f9e4d728ef27c32ada9c467 2023-04-25 06:56:09,935 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a5891719067d4954be47149921ae4512 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a5891719067d4954be47149921ae4512 2023-04-25 06:56:09,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9764ecf3a16d40c1b8ef41ed83841abb to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9764ecf3a16d40c1b8ef41ed83841abb 2023-04-25 06:56:09,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7b771d3cb37d49e79ed35170e66edc2b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7b771d3cb37d49e79ed35170e66edc2b 2023-04-25 06:56:09,939 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/14817c7a27c94c22b5955768d2333d68 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/14817c7a27c94c22b5955768d2333d68 2023-04-25 06:56:09,940 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/4e89ec1ca9b7480dbfec9454d5882bad to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/4e89ec1ca9b7480dbfec9454d5882bad 2023-04-25 06:56:09,941 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/349a0471e5574d6484d1854a0d9371f1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/349a0471e5574d6484d1854a0d9371f1 2023-04-25 06:56:09,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/546ae7f060e74cc98c94270b7128d9c7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/546ae7f060e74cc98c94270b7128d9c7 2023-04-25 06:56:09,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/122d347388cb48c1af8d69e24c1cc362 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/122d347388cb48c1af8d69e24c1cc362 2023-04-25 06:56:09,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/ff6a745658174b229de2605d3b40aadc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/ff6a745658174b229de2605d3b40aadc 2023-04-25 06:56:09,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b199729201c241a59005aeef8247bc62 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/b199729201c241a59005aeef8247bc62 2023-04-25 06:56:09,945 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/011848a908a840cbbbfe7da39cb0a9f5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/011848a908a840cbbbfe7da39cb0a9f5 2023-04-25 06:56:09,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/538b79639c2240ec837cf5b771d8dc8d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/538b79639c2240ec837cf5b771d8dc8d 2023-04-25 06:56:09,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f0348d7fe9264b0fb08fec2a161300d6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f0348d7fe9264b0fb08fec2a161300d6 2023-04-25 06:56:09,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/79affef4a8fb4a08a2df04fd24191149 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/79affef4a8fb4a08a2df04fd24191149 2023-04-25 06:56:09,948 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a9142aef74ea4033b8a7896ef3004297 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/a9142aef74ea4033b8a7896ef3004297 2023-04-25 06:56:09,948 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e66733bf365d4e09b61fb01ba9cad021 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e66733bf365d4e09b61fb01ba9cad021 2023-04-25 06:56:09,949 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e7e7f6c533c34039b7a5d33806dd58e4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/e7e7f6c533c34039b7a5d33806dd58e4 2023-04-25 06:56:09,950 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/75595679523b4ad1a341696aa0ab1fca to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/75595679523b4ad1a341696aa0ab1fca 2023-04-25 06:56:09,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9a9d9fcbaaca48e4a4b63c4ed714751d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/9a9d9fcbaaca48e4a4b63c4ed714751d 2023-04-25 06:56:09,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/51aed01c1a914002ac229f715eb045ed to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/51aed01c1a914002ac229f715eb045ed 2023-04-25 06:56:09,952 DEBUG [StoreCloser-TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7c0d44eb6ca5467ea7381616623b2fa8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/7c0d44eb6ca5467ea7381616623b2fa8 2023-04-25 06:56:09,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/recovered.edits/494.seqid, newMaxSeqId=494, maxSeqId=1 2023-04-25 06:56:09,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4. 2023-04-25 06:56:09,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 75fc43298851a5c4f1060516f9ac39d4: 2023-04-25 06:56:09,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:56:09,959 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=75fc43298851a5c4f1060516f9ac39d4, regionState=CLOSED 2023-04-25 06:56:09,960 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405769959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405769959"}]},"ts":"1682405769959"} 2023-04-25 06:56:09,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-25 06:56:09,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 75fc43298851a5c4f1060516f9ac39d4, server=jenkins-hbase4.apache.org,41021,1682405622759 in 1.5100 sec 2023-04-25 06:56:09,962 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-25 06:56:09,963 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=75fc43298851a5c4f1060516f9ac39d4, UNASSIGN in 1.5130 sec 2023-04-25 06:56:09,963 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405769963"}]},"ts":"1682405769963"} 2023-04-25 06:56:09,964 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 06:56:09,968 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 06:56:09,969 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5260 sec 2023-04-25 06:56:10,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 06:56:10,550 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-25 06:56:10,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 06:56:10,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:10,553 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:10,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-25 06:56:10,553 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:10,557 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:56:10,559 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/recovered.edits] 2023-04-25 06:56:10,561 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37a7b9e7f909499bab2383ed59b9faed to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/37a7b9e7f909499bab2383ed59b9faed 2023-04-25 06:56:10,562 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/98cbaad8dcdb46caaff5813d4cfdad2a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/A/98cbaad8dcdb46caaff5813d4cfdad2a 2023-04-25 06:56:10,565 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ab369359f1994c98bec1f4f341857968 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/ab369359f1994c98bec1f4f341857968 2023-04-25 06:56:10,566 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cdcf8af2798145e69b8ae9a48bcf0aa8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/B/cdcf8af2798145e69b8ae9a48bcf0aa8 2023-04-25 06:56:10,568 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/036fd67146dc4d78bc5a32a43ce58f5d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/036fd67146dc4d78bc5a32a43ce58f5d 2023-04-25 06:56:10,569 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f3617f2fa17f4a008c5155b0634ce6a5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/C/f3617f2fa17f4a008c5155b0634ce6a5 2023-04-25 06:56:10,571 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/recovered.edits/494.seqid to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4/recovered.edits/494.seqid 2023-04-25 06:56:10,572 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/75fc43298851a5c4f1060516f9ac39d4 2023-04-25 06:56:10,572 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:56:10,574 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:10,575 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 06:56:10,576 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 06:56:10,577 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:10,577 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 06:56:10,577 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682405770577"}]},"ts":"9223372036854775807"} 2023-04-25 06:56:10,579 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 06:56:10,579 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 75fc43298851a5c4f1060516f9ac39d4, NAME => 'TestAcidGuarantees,,1682405737882.75fc43298851a5c4f1060516f9ac39d4.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 06:56:10,579 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 06:56:10,579 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682405770579"}]},"ts":"9223372036854775807"} 2023-04-25 06:56:10,581 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 06:56:10,582 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:10,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-25 06:56:10,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-25 06:56:10,654 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-25 06:56:10,665 INFO [Listener at localhost/41663] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=549 (was 549), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=389 (was 385) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=4530 (was 4202) - AvailableMemoryMB LEAK? - 2023-04-25 06:56:10,678 INFO [Listener at localhost/41663] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=549, MaxFileDescriptor=60000, SystemLoadAverage=389, ProcessCount=172, AvailableMemoryMB=4530 2023-04-25 06:56:10,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:56:10,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:56:10,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:10,683 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 06:56:10,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-25 06:56:10,684 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 06:56:10,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 06:56:10,685 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:10,685 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 empty. 2023-04-25 06:56:10,685 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:10,685 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:56:10,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 06:56:10,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 06:56:11,092 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:56:11,094 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e6358a6a069ce34b04034f9d45664d77, NAME => 'TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp 2023-04-25 06:56:11,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 06:56:11,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:56:11,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e6358a6a069ce34b04034f9d45664d77, disabling compactions & flushes 2023-04-25 06:56:11,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. after waiting 0 ms 2023-04-25 06:56:11,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:11,501 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 06:56:11,502 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405771502"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405771502"}]},"ts":"1682405771502"} 2023-04-25 06:56:11,503 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 06:56:11,504 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 06:56:11,504 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405771504"}]},"ts":"1682405771504"} 2023-04-25 06:56:11,504 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 06:56:11,507 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, ASSIGN}] 2023-04-25 06:56:11,508 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, ASSIGN 2023-04-25 06:56:11,509 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=false 2023-04-25 06:56:11,660 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:11,660 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405771660"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405771660"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405771660"}]},"ts":"1682405771660"} 2023-04-25 06:56:11,661 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:56:11,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 06:56:11,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e6358a6a069ce34b04034f9d45664d77, NAME => 'TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:56:11,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:56:11,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,819 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,820 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A 2023-04-25 06:56:11,820 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A 2023-04-25 06:56:11,820 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:56:11,820 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6358a6a069ce34b04034f9d45664d77 columnFamilyName A 2023-04-25 06:56:11,821 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(310): Store=e6358a6a069ce34b04034f9d45664d77/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:56:11,821 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,822 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B 2023-04-25 06:56:11,822 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B 2023-04-25 06:56:11,822 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:56:11,822 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6358a6a069ce34b04034f9d45664d77 columnFamilyName B 2023-04-25 06:56:11,822 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(310): Store=e6358a6a069ce34b04034f9d45664d77/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:56:11,822 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,823 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C 2023-04-25 06:56:11,823 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C 2023-04-25 06:56:11,824 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:56:11,824 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6358a6a069ce34b04034f9d45664d77 columnFamilyName C 2023-04-25 06:56:11,824 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(310): Store=e6358a6a069ce34b04034f9d45664d77/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:56:11,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:56:11,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:11,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 06:56:11,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e6358a6a069ce34b04034f9d45664d77; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63088981, jitterRate=-0.0599009245634079}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:56:11,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:11,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., pid=56, masterSystemTime=1682405771813 2023-04-25 06:56:11,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:11,832 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:11,832 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405771832"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405771832"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405771832"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405771832"}]},"ts":"1682405771832"} 2023-04-25 06:56:11,834 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-25 06:56:11,834 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 in 172 msec 2023-04-25 06:56:11,835 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-25 06:56:11,835 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, ASSIGN in 327 msec 2023-04-25 06:56:11,836 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 06:56:11,836 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405771836"}]},"ts":"1682405771836"} 2023-04-25 06:56:11,836 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 06:56:11,840 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 06:56:11,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-25 06:56:12,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 06:56:12,789 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-25 06:56:12,789 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x050d2537 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:12,794 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54cb8b37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:12,796 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:12,797 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52402, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:12,798 DEBUG [Listener at localhost/41663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 06:56:12,800 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 06:56:12,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 06:56:12,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 06:56:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:13,217 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 06:56:13,218 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 06:56:13,218 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 06:56:13,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 06:56:13,222 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, REOPEN/MOVE}] 2023-04-25 06:56:13,223 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, REOPEN/MOVE 2023-04-25 06:56:13,223 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,223 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405773223"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405773223"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405773223"}]},"ts":"1682405773223"} 2023-04-25 06:56:13,224 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:56:13,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e6358a6a069ce34b04034f9d45664d77, disabling compactions & flushes 2023-04-25 06:56:13,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. after waiting 0 ms 2023-04-25 06:56:13,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,380 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 06:56:13,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:13,381 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e6358a6a069ce34b04034f9d45664d77 to self. 2023-04-25 06:56:13,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,382 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=CLOSED 2023-04-25 06:56:13,382 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405773382"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405773382"}]},"ts":"1682405773382"} 2023-04-25 06:56:13,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-25 06:56:13,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 in 159 msec 2023-04-25 06:56:13,385 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,41021,1682405622759; forceNewPlan=false, retain=true 2023-04-25 06:56:13,535 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,535 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405773535"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405773535"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405773535"}]},"ts":"1682405773535"} 2023-04-25 06:56:13,537 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:56:13,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e6358a6a069ce34b04034f9d45664d77, NAME => 'TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.', STARTKEY => '', ENDKEY => ''} 2023-04-25 06:56:13,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 06:56:13,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,692 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,693 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A 2023-04-25 06:56:13,693 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A 2023-04-25 06:56:13,693 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:56:13,693 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6358a6a069ce34b04034f9d45664d77 columnFamilyName A 2023-04-25 06:56:13,694 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(310): Store=e6358a6a069ce34b04034f9d45664d77/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:56:13,695 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,695 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B 2023-04-25 06:56:13,695 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B 2023-04-25 06:56:13,696 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:56:13,696 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6358a6a069ce34b04034f9d45664d77 columnFamilyName B 2023-04-25 06:56:13,696 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(310): Store=e6358a6a069ce34b04034f9d45664d77/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:56:13,696 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,697 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C 2023-04-25 06:56:13,697 DEBUG [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C 2023-04-25 06:56:13,697 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 06:56:13,697 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6358a6a069ce34b04034f9d45664d77 columnFamilyName C 2023-04-25 06:56:13,698 INFO [StoreOpener-e6358a6a069ce34b04034f9d45664d77-1] regionserver.HStore(310): Store=e6358a6a069ce34b04034f9d45664d77/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 06:56:13,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 06:56:13,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e6358a6a069ce34b04034f9d45664d77; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74662394, jitterRate=0.11255636811256409}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 06:56:13,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:13,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., pid=61, masterSystemTime=1682405773688 2023-04-25 06:56:13,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,705 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,705 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405773705"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682405773705"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682405773705"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682405773705"}]},"ts":"1682405773705"} 2023-04-25 06:56:13,707 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-25 06:56:13,707 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 in 169 msec 2023-04-25 06:56:13,708 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-25 06:56:13,708 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, REOPEN/MOVE in 485 msec 2023-04-25 06:56:13,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-25 06:56:13,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-25 06:56:13,711 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 908 msec 2023-04-25 06:56:13,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-25 06:56:13,712 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x68d563c7 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,718 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41dc777, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,718 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4b13e8aa to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,722 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@89b6154, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,722 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x55d0f313 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,726 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d207bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,726 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x09d61148 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,729 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5bcc330a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,730 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x1a4acea2 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,736 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23afc0f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,736 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x0f1adb60 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,740 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@526f2221, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,741 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4ed73d5e to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,745 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17cf2e3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,745 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x43ea1da2 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,753 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3420e54c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,753 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x35c3cca2 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,758 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26d7d0b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,758 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(139): Connect 0x4b1973b9 to 127.0.0.1:60416 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 06:56:13,762 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@601410a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 06:56:13,771 DEBUG [hconnection-0x722c90ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,772 DEBUG [hconnection-0x772c87b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,774 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:13,776 DEBUG [hconnection-0x54344354-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,777 DEBUG [hconnection-0x3eda7df8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,777 DEBUG [hconnection-0x1df3c017-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,777 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39888, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,786 DEBUG [hconnection-0x1602ec74-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,786 DEBUG [hconnection-0xb237459-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,786 DEBUG [hconnection-0x2ab975c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,786 DEBUG [hconnection-0x556eb9ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,786 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39894, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,789 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39920, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,789 DEBUG [hconnection-0x442268b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 06:56:13,789 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,790 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,791 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39904, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,791 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,791 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39938, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:13,797 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 06:56:13,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:13,797 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:13,798 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:13,798 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:13,798 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:13,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:13,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:13,800 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,800 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:13,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:13,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,801 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:13,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:13,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:13,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:13,804 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:13,804 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:13,805 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 06:56:13,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:13,805 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:13,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:13,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:13,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:13,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:13,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:13,807 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:13,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405833852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405773851, max timestamp=1682405773855Num uniques 859; 2023-04-25 06:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405773851, max timestamp=1682405773855Num uniques 859; 2023-04-25 06:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682405773851, max timestamp=1682405773855Num uniques 859; 2023-04-25 06:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 06:56:13,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405833868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405833869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405833870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405833871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405833958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405833972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405833973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405833973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:13,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:13,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405833974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405834160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405834175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405834176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405834176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405834177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,226 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042507ea889148d54a16a08f0312bba56a5d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042507ea889148d54a16a08f0312bba56a5d 2023-04-25 06:56:14,232 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/9adaa40bfab542f299e93373d9a29de4 2023-04-25 06:56:14,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405834462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405834478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405834479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405834480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:14,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405834480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,642 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/9dbbfb0ecac74d62bc798bc551870051 2023-04-25 06:56:14,652 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/d4b0be4a0d3b4841991fcea4c0079eac 2023-04-25 06:56:14,656 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/9adaa40bfab542f299e93373d9a29de4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9adaa40bfab542f299e93373d9a29de4 2023-04-25 06:56:14,659 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9adaa40bfab542f299e93373d9a29de4, entries=150, sequenceid=15, filesize=25.1 K 2023-04-25 06:56:14,660 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/9dbbfb0ecac74d62bc798bc551870051 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9dbbfb0ecac74d62bc798bc551870051 2023-04-25 06:56:14,663 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9dbbfb0ecac74d62bc798bc551870051, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 06:56:14,664 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/d4b0be4a0d3b4841991fcea4c0079eac as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d4b0be4a0d3b4841991fcea4c0079eac 2023-04-25 06:56:14,667 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d4b0be4a0d3b4841991fcea4c0079eac, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 06:56:14,669 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e6358a6a069ce34b04034f9d45664d77 in 863ms, sequenceid=15, compaction requested=false 2023-04-25 06:56:14,669 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:14,669 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:14,670 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:14,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:14,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa237c8[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:14,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:14,672 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,678 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:14,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,679 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:14,680 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:14,683 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:14,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:14,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:14,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:14,683 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:14,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:14,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:14,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:14,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,687 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:14,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:14,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:14,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:14,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:14,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:14,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:14,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,691 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,691 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:14,691 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:14,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:14,791 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:14,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:14,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:14,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:14,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:14,796 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:14,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:14,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:14,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:14,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,798 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:14,798 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:14,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:14,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:14,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:14,800 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:14,800 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:14,800 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-25 06:56:14,800 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:14,801 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:14,801 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:14,801 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:14,801 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:14,801 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:14,820 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425b528e37dadaf43559a2bcf57397dd587 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b528e37dadaf43559a2bcf57397dd587 2023-04-25 06:56:14,826 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/6fd4c8789c2d46658d23d32318cae002 2023-04-25 06:56:14,837 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/a41c05d64cf8476b83056f0a16998e49 2023-04-25 06:56:14,846 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/dbdb692e10754cec91156ba9a9577524 2023-04-25 06:56:14,849 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/6fd4c8789c2d46658d23d32318cae002 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6fd4c8789c2d46658d23d32318cae002 2023-04-25 06:56:14,852 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6fd4c8789c2d46658d23d32318cae002, entries=150, sequenceid=51, filesize=25.1 K 2023-04-25 06:56:14,853 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/a41c05d64cf8476b83056f0a16998e49 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a41c05d64cf8476b83056f0a16998e49 2023-04-25 06:56:14,856 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a41c05d64cf8476b83056f0a16998e49, entries=150, sequenceid=51, filesize=11.6 K 2023-04-25 06:56:14,857 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/dbdb692e10754cec91156ba9a9577524 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dbdb692e10754cec91156ba9a9577524 2023-04-25 06:56:14,868 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dbdb692e10754cec91156ba9a9577524, entries=150, sequenceid=51, filesize=11.6 K 2023-04-25 06:56:14,869 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for e6358a6a069ce34b04034f9d45664d77 in 69ms, sequenceid=51, compaction requested=false 2023-04-25 06:56:14,869 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:14,869 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:14,871 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,871 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:14,871 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,871 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377c0f9d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:14,873 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:14,873 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,874 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:14,874 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,874 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:14,875 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:14,875 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:14,876 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:14,876 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:14,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:14,879 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:14,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:14,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:14,879 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:14,879 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:14,881 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,881 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:14,881 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,881 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:14,883 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:14,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:14,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:14,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:14,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:14,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:14,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:14,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:14,885 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:14,885 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:14,885 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:14,886 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,885 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:14,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:14,986 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:14,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:14,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:14,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:14,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:14,990 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:14,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:14,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:14,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:14,991 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,991 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:14,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:14,992 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:14,993 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:14,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:14,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:14,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:14,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:14,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:14,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:14,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:14,994 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:14,994 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:14,994 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 06:56:14,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:14,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:14,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:14,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:14,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:14,995 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:14,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:14,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:15,008 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f4470a6368d041f4a12ffafe9370cf06 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f4470a6368d041f4a12ffafe9370cf06 2023-04-25 06:56:15,016 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/73ac22131f7249ec9dde9723f474097a 2023-04-25 06:56:15,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405775010, max timestamp=1682405775017Num uniques 859; 2023-04-25 06:56:15,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:56:15,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405775010, max timestamp=1682405775017Num uniques 859; 2023-04-25 06:56:15,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682405775010, max timestamp=1682405775017Num uniques 859; 2023-04-25 06:56:15,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:56:15,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 06:56:15,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405835034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405835034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405835035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405835036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405835039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405835140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405835140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405835140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405835141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405835143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405835344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405835344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405835345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405835345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405835346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,431 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/2efa23b4d6064025b5e375f5d44fdccf 2023-04-25 06:56:15,441 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/e7677a17fee24b91b549ce31c6e91c5f 2023-04-25 06:56:15,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/73ac22131f7249ec9dde9723f474097a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/73ac22131f7249ec9dde9723f474097a 2023-04-25 06:56:15,448 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/73ac22131f7249ec9dde9723f474097a, entries=300, sequenceid=60, filesize=45.5 K 2023-04-25 06:56:15,449 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/2efa23b4d6064025b5e375f5d44fdccf as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/2efa23b4d6064025b5e375f5d44fdccf 2023-04-25 06:56:15,452 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/2efa23b4d6064025b5e375f5d44fdccf, entries=150, sequenceid=60, filesize=11.6 K 2023-04-25 06:56:15,453 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/e7677a17fee24b91b549ce31c6e91c5f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/e7677a17fee24b91b549ce31c6e91c5f 2023-04-25 06:56:15,456 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/e7677a17fee24b91b549ce31c6e91c5f, entries=150, sequenceid=60, filesize=11.6 K 2023-04-25 06:56:15,457 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for e6358a6a069ce34b04034f9d45664d77 in 463ms, sequenceid=60, compaction requested=true 2023-04-25 06:56:15,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:15,457 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:15,459 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,459 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,459 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,459 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@626da48c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:15,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:15,460 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:15,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:15,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:15,465 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:15,467 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,467 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:15,467 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,467 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:15,469 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:15,469 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:15,469 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:15,470 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:15,470 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:15,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,471 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,472 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,473 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:15,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:15,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:15,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,476 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,476 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:15,476 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:15,476 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:15,488 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 06:56:15,577 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:15,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:15,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:15,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:15,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:15,585 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:15,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:15,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:15,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:15,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,587 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,587 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:15,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:15,588 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,588 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:15,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:15,589 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:15,590 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:15,590 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:15,590 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:15,590 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:15,590 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:15,590 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:15,590 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-25 06:56:15,590 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:15,590 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:15,591 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:15,591 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:15,591 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:15,591 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:15,619 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d9d1a910e56d4c40b1e0b78ca6e53a80 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d9d1a910e56d4c40b1e0b78ca6e53a80 2023-04-25 06:56:15,626 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/acc27e0dbf5d4e8eb0e198c97fd112a4 2023-04-25 06:56:15,643 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/6bfe350ddf1c434fadb24721ecffd856 2023-04-25 06:56:15,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:15,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:15,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405835653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405835654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405835655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405835656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405835654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,674 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/1b010923088748868aa3fe2c5146c59d 2023-04-25 06:56:15,679 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/acc27e0dbf5d4e8eb0e198c97fd112a4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/acc27e0dbf5d4e8eb0e198c97fd112a4 2023-04-25 06:56:15,684 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/acc27e0dbf5d4e8eb0e198c97fd112a4, entries=150, sequenceid=97, filesize=25.1 K 2023-04-25 06:56:15,685 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/6bfe350ddf1c434fadb24721ecffd856 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6bfe350ddf1c434fadb24721ecffd856 2023-04-25 06:56:15,692 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6bfe350ddf1c434fadb24721ecffd856, entries=150, sequenceid=97, filesize=11.6 K 2023-04-25 06:56:15,693 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/1b010923088748868aa3fe2c5146c59d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1b010923088748868aa3fe2c5146c59d 2023-04-25 06:56:15,698 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1b010923088748868aa3fe2c5146c59d, entries=150, sequenceid=97, filesize=11.6 K 2023-04-25 06:56:15,699 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e6358a6a069ce34b04034f9d45664d77 in 108ms, sequenceid=97, compaction requested=true 2023-04-25 06:56:15,699 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:15,699 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:15,700 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:15,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31757d0f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:15,702 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:15,702 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,703 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,704 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:15,704 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:15,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,705 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:15,705 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,705 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:15,708 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:15,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:15,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:15,708 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:15,708 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:15,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,711 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,711 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,713 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,716 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:15,716 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:15,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:15,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:15,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:15,717 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:15,717 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:15,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:15,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:15,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:56:15,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:15,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:15,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:15,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:15,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:15,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:15,781 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c768fdceaf204f8eaf448ca9471f01e8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c768fdceaf204f8eaf448ca9471f01e8 2023-04-25 06:56:15,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405835784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405835786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405835787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405835789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405835791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,817 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:15,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:15,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:15,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:15,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:15,820 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:15,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:15,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:15,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:15,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,821 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:15,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:15,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,822 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:15,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:15,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:15,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:15,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:15,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:15,824 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:15,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,825 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,826 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:15,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a12109e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:15,827 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:15,827 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,829 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:15,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:15,829 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:15,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,830 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:15,831 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:15,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:15,833 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:15,833 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:15,833 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:15,833 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:15,833 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:15,835 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,835 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,835 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,835 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,836 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,838 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:15,838 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:15,838 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,839 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:15,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:15,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:15,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,839 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:15,839 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:15,839 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:15,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405835892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405835893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405835893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405835896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:15,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405835896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,939 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:15,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:15,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:15,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:15,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:15,943 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:15,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:15,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:15,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:15,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,944 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,946 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,947 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:15,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:15,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:15,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:15,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:15,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:15,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:15,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:15,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:15,949 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:15,949 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:15,949 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:15,949 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:15,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:15,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:15,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:15,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:15,950 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc5fea6[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:15,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:15,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,954 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:15,954 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,954 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:15,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:15,954 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:15,955 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:15,955 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,955 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:15,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:15,958 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:15,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:15,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:15,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:15,958 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:15,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:15,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,964 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:15,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,964 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,966 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:15,968 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:15,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:15,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:15,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:15,968 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:15,969 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:15,969 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:15,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:15,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:15,969 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,069 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:16,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:16,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:16,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:16,072 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:16,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:16,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:16,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,073 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,073 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:16,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:16,075 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,075 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:16,075 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:16,075 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:16,077 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12036056[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:16,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:16,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,083 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:16,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:16,083 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:16,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,083 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:16,085 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,085 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:16,087 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:16,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:16,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:16,087 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:16,087 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:16,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,090 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,091 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,092 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,094 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,094 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:16,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:16,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:16,095 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:16,095 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:16,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405836095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405836096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405836096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405836098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405836099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,188 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/19fce01a35fe43d091a2bcd30645b07f 2023-04-25 06:56:16,195 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:16,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:16,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:16,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:16,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:16,198 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:16,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:16,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:16,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/9436a73d3ca542da9940c639076d5f6e 2023-04-25 06:56:16,200 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,201 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,201 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,201 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:16,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:16,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:16,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:16,202 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:16,202 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,202 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:16,203 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,203 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:16,203 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:16,203 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:16,205 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,205 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@804fa1c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:16,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:16,208 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,209 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,209 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:16,209 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:16,209 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:16,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,212 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/d60a93816372438baf9f872a15b1fc85 2023-04-25 06:56:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:16,216 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:16,216 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:16,216 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:16,216 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:16,216 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:16,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/19fce01a35fe43d091a2bcd30645b07f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/19fce01a35fe43d091a2bcd30645b07f 2023-04-25 06:56:16,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,219 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,220 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/19fce01a35fe43d091a2bcd30645b07f, entries=150, sequenceid=109, filesize=25.1 K 2023-04-25 06:56:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/9436a73d3ca542da9940c639076d5f6e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9436a73d3ca542da9940c639076d5f6e 2023-04-25 06:56:16,231 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,231 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:16,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:16,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,232 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,232 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:16,232 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:16,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,232 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:16,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9436a73d3ca542da9940c639076d5f6e, entries=150, sequenceid=109, filesize=11.6 K 2023-04-25 06:56:16,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/d60a93816372438baf9f872a15b1fc85 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d60a93816372438baf9f872a15b1fc85 2023-04-25 06:56:16,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d60a93816372438baf9f872a15b1fc85, entries=150, sequenceid=109, filesize=11.6 K 2023-04-25 06:56:16,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e6358a6a069ce34b04034f9d45664d77 in 486ms, sequenceid=109, compaction requested=true 2023-04-25 06:56:16,250 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:16,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:16,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:16,250 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:56:16,250 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 06:56:16,251 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 149554 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:56:16,251 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/A is initiating minor compaction (all files) 2023-04-25 06:56:16,251 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/A in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:16,252 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 06:56:16,252 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/B is initiating minor compaction (all files) 2023-04-25 06:56:16,252 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/B in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,252 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9dbbfb0ecac74d62bc798bc551870051, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a41c05d64cf8476b83056f0a16998e49, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/2efa23b4d6064025b5e375f5d44fdccf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6bfe350ddf1c434fadb24721ecffd856, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9436a73d3ca542da9940c639076d5f6e] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=58.0 K 2023-04-25 06:56:16,252 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9adaa40bfab542f299e93373d9a29de4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6fd4c8789c2d46658d23d32318cae002, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/73ac22131f7249ec9dde9723f474097a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/acc27e0dbf5d4e8eb0e198c97fd112a4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/19fce01a35fe43d091a2bcd30645b07f] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=146.0 K 2023-04-25 06:56:16,252 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9dbbfb0ecac74d62bc798bc551870051, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682405773800 2023-04-25 06:56:16,253 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9adaa40bfab542f299e93373d9a29de4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682405773800 2023-04-25 06:56:16,253 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a41c05d64cf8476b83056f0a16998e49, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682405773863 2023-04-25 06:56:16,253 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6fd4c8789c2d46658d23d32318cae002, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682405773863 2023-04-25 06:56:16,253 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 2efa23b4d6064025b5e375f5d44fdccf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682405774988 2023-04-25 06:56:16,253 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 73ac22131f7249ec9dde9723f474097a, keycount=300, bloomtype=ROW, size=45.5 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682405774986 2023-04-25 06:56:16,253 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6bfe350ddf1c434fadb24721ecffd856, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682405775032 2023-04-25 06:56:16,254 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting acc27e0dbf5d4e8eb0e198c97fd112a4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682405775032 2023-04-25 06:56:16,254 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9436a73d3ca542da9940c639076d5f6e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682405775652 2023-04-25 06:56:16,254 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 19fce01a35fe43d091a2bcd30645b07f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682405775652 2023-04-25 06:56:16,263 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#B#compaction#485 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:16,273 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#A#compaction#486 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:16,332 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:16,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:16,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:16,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:16,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:16,336 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:16,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:16,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:16,336 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,337 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,338 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,338 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:16,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:16,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:16,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:16,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:16,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:16,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,340 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:16,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:16,341 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:16,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:16,341 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:56:16,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:16,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:16,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:16,341 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:16,342 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:16,342 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:16,357 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042574c10ca700f0438c8f3756a925470d9c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042574c10ca700f0438c8f3756a925470d9c 2023-04-25 06:56:16,362 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=129, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/93550532eb9148cea4ae6058bca98462 2023-04-25 06:56:16,372 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/6c9a6a14f5194cf4994b6022f9a47b28 2023-04-25 06:56:16,386 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/1effbfe44fe64c9d8fd4076a58315804 2023-04-25 06:56:16,390 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/93550532eb9148cea4ae6058bca98462 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/93550532eb9148cea4ae6058bca98462 2023-04-25 06:56:16,393 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/93550532eb9148cea4ae6058bca98462, entries=150, sequenceid=129, filesize=25.1 K 2023-04-25 06:56:16,394 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/6c9a6a14f5194cf4994b6022f9a47b28 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6c9a6a14f5194cf4994b6022f9a47b28 2023-04-25 06:56:16,397 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6c9a6a14f5194cf4994b6022f9a47b28, entries=150, sequenceid=129, filesize=11.6 K 2023-04-25 06:56:16,398 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/1effbfe44fe64c9d8fd4076a58315804 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1effbfe44fe64c9d8fd4076a58315804 2023-04-25 06:56:16,417 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1effbfe44fe64c9d8fd4076a58315804, entries=150, sequenceid=129, filesize=11.6 K 2023-04-25 06:56:16,418 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for e6358a6a069ce34b04034f9d45664d77 in 77ms, sequenceid=129, compaction requested=true 2023-04-25 06:56:16,418 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:16,418 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:16,419 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,420 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52489d62[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:16,422 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:16,422 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,424 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:16,424 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,424 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:16,426 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,426 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,426 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,426 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:16,429 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:16,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:16,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:16,429 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:16,429 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:16,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,431 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,438 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,438 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:16,438 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,438 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,438 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,439 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,439 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,438 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,439 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:16,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:16,439 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,439 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:16,439 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:16,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:16,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 06:56:16,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:16,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:16,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:16,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:16,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:16,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:16,466 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042528d7b280e47e4b67a9c002b630d3dafc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042528d7b280e47e4b67a9c002b630d3dafc 2023-04-25 06:56:16,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682405776470, max timestamp=1682405776503Num uniques 859; 2023-04-25 06:56:16,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 06:56:16,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682405776470, max timestamp=1682405776503Num uniques 859; 2023-04-25 06:56:16,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682405776470, max timestamp=1682405776503Num uniques 859; 2023-04-25 06:56:16,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 06:56:16,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 06:56:16,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405836516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405836516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405836517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405836521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405836522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,539 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:16,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:16,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:16,543 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:16,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:16,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:16,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,543 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,544 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,544 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,545 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,545 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:16,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:16,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:16,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:16,547 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:16,548 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:16,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ca6943[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:16,549 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:16,550 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:16,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:16,553 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:16,554 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,554 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:16,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:16,556 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:16,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:16,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:16,556 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:16,556 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:16,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,557 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,559 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,561 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,562 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,562 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:16,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:16,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:16,563 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:16,563 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:16,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405836623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405836624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405836624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405836627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405836627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,663 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:16,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:16,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:16,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:16,667 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:16,667 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,667 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:16,667 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:16,667 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,667 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,668 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,668 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:16,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:16,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:16,670 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,670 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:16,670 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:16,671 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:16,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:16,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:16,671 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:16,673 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,673 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,674 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:16,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c2a4dc[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:16,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:16,675 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,676 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:16,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:16,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:16,677 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:16,679 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:16,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:16,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:16,682 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:16,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:16,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:16,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:16,682 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:16,682 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/a8b023463a224282bab1c0711a76e737 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a8b023463a224282bab1c0711a76e737 2023-04-25 06:56:16,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,688 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/B of e6358a6a069ce34b04034f9d45664d77 into a8b023463a224282bab1c0711a76e737(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:16,688 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:16,688 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/B, priority=11, startTime=1682405776250; duration=0sec 2023-04-25 06:56:16,688 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:16,688 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:56:16,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:16,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:16,690 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:16,690 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:16,689 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,690 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,691 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:56:16,691 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/C is initiating minor compaction (all files) 2023-04-25 06:56:16,691 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/C in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:16,691 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d4b0be4a0d3b4841991fcea4c0079eac, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dbdb692e10754cec91156ba9a9577524, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/e7677a17fee24b91b549ce31c6e91c5f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1b010923088748868aa3fe2c5146c59d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d60a93816372438baf9f872a15b1fc85, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1effbfe44fe64c9d8fd4076a58315804] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=69.6 K 2023-04-25 06:56:16,693 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting d4b0be4a0d3b4841991fcea4c0079eac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682405773800 2023-04-25 06:56:16,693 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting dbdb692e10754cec91156ba9a9577524, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682405773863 2023-04-25 06:56:16,693 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e7677a17fee24b91b549ce31c6e91c5f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682405774988 2023-04-25 06:56:16,694 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1b010923088748868aa3fe2c5146c59d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682405775032 2023-04-25 06:56:16,694 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting d60a93816372438baf9f872a15b1fc85, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682405775652 2023-04-25 06:56:16,694 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1effbfe44fe64c9d8fd4076a58315804, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682405775783 2023-04-25 06:56:16,701 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/7496170aac38474fa48bf138e06f7200 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7496170aac38474fa48bf138e06f7200 2023-04-25 06:56:16,706 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#C#compaction#491 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:16,709 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/A of e6358a6a069ce34b04034f9d45664d77 into 7496170aac38474fa48bf138e06f7200(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:16,710 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:16,710 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/A, priority=11, startTime=1682405776250; duration=0sec 2023-04-25 06:56:16,710 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:16,714 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/ea5230e047144784b4f9e4bae37a2a6b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/ea5230e047144784b4f9e4bae37a2a6b 2023-04-25 06:56:16,717 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/C of e6358a6a069ce34b04034f9d45664d77 into ea5230e047144784b4f9e4bae37a2a6b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:16,718 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:16,718 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/C, priority=10, startTime=1682405776250; duration=0sec 2023-04-25 06:56:16,718 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:16,790 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:16,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:16,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:16,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:16,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:16,794 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:16,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:16,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:16,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,794 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,795 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,795 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,796 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:16,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:16,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:16,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:16,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:16,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,798 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:16,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:16,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:16,802 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:16,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:16,803 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:16,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:16,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b977e2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:16,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:16,806 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:16,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:16,808 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:16,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:16,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:16,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:16,816 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:16,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:16,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:16,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:16,816 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:16,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,818 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,819 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:16,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:16,825 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,825 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:16,825 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:16,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405836828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405836828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405836829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405836832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:16,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405836833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,872 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/79831d8db1394d92b6c4ffa36265e4ef 2023-04-25 06:56:16,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e7adfc08dac349c49987d43d9309e0ee 2023-04-25 06:56:16,925 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:16,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:16,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:16,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:16,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:16,929 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:16,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:16,929 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:16,930 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,930 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,932 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:16,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:16,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:16,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:16,933 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:16,933 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:16,934 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:16,941 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:16,941 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:16,941 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:16,941 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:16,944 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500c51da[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:16,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:16,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:16,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,948 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:16,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,949 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,950 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:16,953 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:16,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:16,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:16,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:16,953 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:16,955 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,956 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,956 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:16,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:16,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:16,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,958 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:16,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:16,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:16,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:16,961 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:16,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:16,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:16,962 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:16,962 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:16,961 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:16,962 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:16,962 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:16,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:16,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:17,062 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:17,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:17,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:17,066 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:17,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:17,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:17,067 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,067 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,068 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:17,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:17,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:17,070 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,070 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:17,070 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:17,071 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:17,072 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633faae9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:17,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:17,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:17,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:17,077 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:17,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:17,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:17,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:17,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,078 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:17,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:17,082 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:17,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:17,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:17,082 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:17,082 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:17,083 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,084 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:17,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,084 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,086 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:17,089 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:17,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:17,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:17,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:17,089 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:17,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:17,090 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:17,090 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:17,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405837133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405837133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405837134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405837134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405837140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,190 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:17,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:17,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:17,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:17,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:17,194 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:17,194 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,194 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:17,195 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:17,195 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,195 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,197 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,197 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:17,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:17,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,198 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:17,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:17,199 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,199 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:17,200 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:17,200 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:17,201 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,201 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f271616[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:17,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:17,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,204 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,205 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:17,205 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:17,205 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:17,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:17,206 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:17,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,206 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:17,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:17,210 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:17,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:17,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:17,210 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:17,210 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:17,212 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,212 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,212 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:17,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:17,213 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,214 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:17,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:17,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,217 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,217 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,217 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:17,217 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:17,317 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:17,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/4b4ccb4429884d899dec3912c336bd7e 2023-04-25 06:56:17,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:17,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:17,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:17,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:17,322 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:17,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:17,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:17,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/79831d8db1394d92b6c4ffa36265e4ef as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/79831d8db1394d92b6c4ffa36265e4ef 2023-04-25 06:56:17,324 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:17,324 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,325 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:17,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:17,328 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/79831d8db1394d92b6c4ffa36265e4ef, entries=400, sequenceid=140, filesize=59.4 K 2023-04-25 06:56:17,329 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e7adfc08dac349c49987d43d9309e0ee as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e7adfc08dac349c49987d43d9309e0ee 2023-04-25 06:56:17,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:17,330 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:17,330 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,330 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:17,331 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,331 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:17,331 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:17,331 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:17,332 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:17,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e7adfc08dac349c49987d43d9309e0ee, entries=150, sequenceid=140, filesize=11.8 K 2023-04-25 06:56:17,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/4b4ccb4429884d899dec3912c336bd7e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/4b4ccb4429884d899dec3912c336bd7e 2023-04-25 06:56:17,337 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,337 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ac7477[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:17,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:17,341 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/4b4ccb4429884d899dec3912c336bd7e, entries=150, sequenceid=140, filesize=11.8 K 2023-04-25 06:56:17,342 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:17,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,342 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:17,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e6358a6a069ce34b04034f9d45664d77 in 889ms, sequenceid=140, compaction requested=true 2023-04-25 06:56:17,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:17,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:17,342 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:17,342 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:17,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:17,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:17,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:17,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:17,343 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:17,343 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 112518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:17,344 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/A is initiating minor compaction (all files) 2023-04-25 06:56:17,344 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/A in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,344 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7496170aac38474fa48bf138e06f7200, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/93550532eb9148cea4ae6058bca98462, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/79831d8db1394d92b6c4ffa36265e4ef] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=109.9 K 2023-04-25 06:56:17,344 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35980 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:17,344 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/B is initiating minor compaction (all files) 2023-04-25 06:56:17,344 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/B in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,344 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a8b023463a224282bab1c0711a76e737, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6c9a6a14f5194cf4994b6022f9a47b28, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e7adfc08dac349c49987d43d9309e0ee] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=35.1 K 2023-04-25 06:56:17,344 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,345 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a8b023463a224282bab1c0711a76e737, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682405775652 2023-04-25 06:56:17,345 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 7496170aac38474fa48bf138e06f7200, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682405775652 2023-04-25 06:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,345 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6c9a6a14f5194cf4994b6022f9a47b28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682405775783 2023-04-25 06:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,346 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 93550532eb9148cea4ae6058bca98462, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682405775783 2023-04-25 06:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,346 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 79831d8db1394d92b6c4ffa36265e4ef, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682405776437 2023-04-25 06:56:17,346 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e7adfc08dac349c49987d43d9309e0ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682405776441 2023-04-25 06:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:17,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:17,348 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:17,348 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:17,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:17,349 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:17,349 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:17,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,350 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:17,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:17,351 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,354 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#B#compaction#495 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:17,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,356 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,359 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#A#compaction#494 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:17,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:17,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:17,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:17,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:17,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:17,363 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,362 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:17,362 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:17,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:17,363 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:17,363 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:17,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:17,364 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/8607862ed43b4a819e4b2dda3ea4b57e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/8607862ed43b4a819e4b2dda3ea4b57e 2023-04-25 06:56:17,371 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/B of e6358a6a069ce34b04034f9d45664d77 into 8607862ed43b4a819e4b2dda3ea4b57e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:17,371 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:17,371 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/B, priority=13, startTime=1682405777342; duration=0sec 2023-04-25 06:56:17,371 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:17,371 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-25 06:56:17,372 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-25 06:56:17,372 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-25 06:56:17,372 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. because compaction request was cancelled 2023-04-25 06:56:17,463 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:17,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:17,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:17,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:17,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:17,466 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:17,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:17,466 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:17,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,468 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,468 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:17,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:17,468 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,468 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:17,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:17,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:17,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:17,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:17,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:17,469 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:17,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:17,470 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:17,470 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:17,470 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:17,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:17,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:17,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405837644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405837645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405837646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405837646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405837647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405837748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405837748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405837749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405837750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,782 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/a0a095c88d7e406c9b77aa61792a0fed as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a0a095c88d7e406c9b77aa61792a0fed 2023-04-25 06:56:17,787 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/A of e6358a6a069ce34b04034f9d45664d77 into a0a095c88d7e406c9b77aa61792a0fed(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:17,787 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:17,787 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/A, priority=13, startTime=1682405777342; duration=0sec 2023-04-25 06:56:17,787 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:17,885 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304259e3d022d60164426bf1b1dbc3bb36fea to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259e3d022d60164426bf1b1dbc3bb36fea 2023-04-25 06:56:17,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405837951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405837952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405837953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:17,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:17,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405837956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405838254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405838254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405838257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405838259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,292 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=177, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/b781ebece9cb46e7a8db3369574ca0d3 2023-04-25 06:56:18,311 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/ed9fe92af43c4b348e0466bf9ca0cb6c 2023-04-25 06:56:18,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405838651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,722 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/dc9140e428e548f7a0c2b9e18f9e81d7 2023-04-25 06:56:18,725 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/b781ebece9cb46e7a8db3369574ca0d3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b781ebece9cb46e7a8db3369574ca0d3 2023-04-25 06:56:18,729 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b781ebece9cb46e7a8db3369574ca0d3, entries=150, sequenceid=177, filesize=25.3 K 2023-04-25 06:56:18,729 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/ed9fe92af43c4b348e0466bf9ca0cb6c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/ed9fe92af43c4b348e0466bf9ca0cb6c 2023-04-25 06:56:18,732 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/ed9fe92af43c4b348e0466bf9ca0cb6c, entries=150, sequenceid=177, filesize=11.8 K 2023-04-25 06:56:18,733 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/dc9140e428e548f7a0c2b9e18f9e81d7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dc9140e428e548f7a0c2b9e18f9e81d7 2023-04-25 06:56:18,736 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dc9140e428e548f7a0c2b9e18f9e81d7, entries=150, sequenceid=177, filesize=11.8 K 2023-04-25 06:56:18,737 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for e6358a6a069ce34b04034f9d45664d77 in 1267ms, sequenceid=177, compaction requested=true 2023-04-25 06:56:18,737 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:18,737 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:18,739 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,739 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@201c6579[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:18,740 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:18,741 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,742 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,742 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:18,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,742 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:18,742 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:18,742 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:18,743 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,743 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:18,743 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,744 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:18,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:18,746 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:18,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:18,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:18,746 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:18,746 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:18,749 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,749 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,749 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:18,749 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,751 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:18,753 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:18,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:18,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:18,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,753 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:18,753 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,753 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:18,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:18,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:18,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:56:18,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:18,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:18,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:18,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:18,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:18,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:18,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405838781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405838782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405838783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405838785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,853 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:18,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:18,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:18,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:18,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:18,857 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:18,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:18,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:18,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:18,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,857 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,859 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:18,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:18,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:18,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:18,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:18,861 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:18,861 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:18,861 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:18,861 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:18,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:18,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6afb71[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:18,864 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:18,864 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,867 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:18,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:18,868 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:18,870 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:18,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:18,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:18,871 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:18,871 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:18,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:18,872 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,873 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,874 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,876 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:18,876 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:18,876 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,876 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,876 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,876 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:18,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:18,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:18,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:18,876 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,877 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,877 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,877 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:18,877 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:18,877 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:18,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405838887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405838889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405838889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:18,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405838895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,977 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:18,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:18,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:18,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:18,981 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:18,981 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:18,981 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:18,981 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:18,981 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,981 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,984 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:18,984 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:18,984 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,984 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:18,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:18,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:18,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:18,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:18,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:18,986 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:18,987 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,988 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,988 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:18,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19dadd0c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:18,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:18,990 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,994 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:18,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:18,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:18,997 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:18,997 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:18,997 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:18,999 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:18,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:18,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:18,999 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:18,999 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,001 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,002 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,003 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,006 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,006 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,006 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,006 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405839091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405839092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405839092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405839096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,106 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:19,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:19,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:19,110 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:19,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:19,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:19,111 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,111 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,112 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,112 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,112 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,112 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:19,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:19,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:19,118 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,118 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,119 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:19,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68240d8c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:19,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:19,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,122 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:19,122 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,122 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:19,123 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,123 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,123 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:19,126 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:19,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:19,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:19,126 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:19,127 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,128 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,129 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,130 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,134 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,134 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,134 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,134 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,200 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c954e9220d7b4e89b6866ad75c79f663 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c954e9220d7b4e89b6866ad75c79f663 2023-04-25 06:56:19,207 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/ad084ccca90844a793f4d32c26aa4797 2023-04-25 06:56:19,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/0112556ad06e4f04a24d644d48f9dcbe 2023-04-25 06:56:19,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/3e2575af41274d90a381d23d11902c1d 2023-04-25 06:56:19,234 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:19,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:19,236 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/ad084ccca90844a793f4d32c26aa4797 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/ad084ccca90844a793f4d32c26aa4797 2023-04-25 06:56:19,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:19,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:19,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:19,239 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:19,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:19,239 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:19,240 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,240 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,241 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,241 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,242 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:19,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:19,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/ad084ccca90844a793f4d32c26aa4797, entries=200, sequenceid=190, filesize=32.1 K 2023-04-25 06:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,246 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:19,246 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:19,246 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:19,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/0112556ad06e4f04a24d644d48f9dcbe as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/0112556ad06e4f04a24d644d48f9dcbe 2023-04-25 06:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,247 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b509a5c[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:19,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:19,250 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,250 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/0112556ad06e4f04a24d644d48f9dcbe, entries=150, sequenceid=190, filesize=11.8 K 2023-04-25 06:56:19,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:19,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/3e2575af41274d90a381d23d11902c1d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3e2575af41274d90a381d23d11902c1d 2023-04-25 06:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:19,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3e2575af41274d90a381d23d11902c1d, entries=150, sequenceid=190, filesize=11.8 K 2023-04-25 06:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,256 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e6358a6a069ce34b04034f9d45664d77 in 494ms, sequenceid=190, compaction requested=true 2023-04-25 06:56:19,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,256 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:19,256 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:19,256 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:19,256 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,257 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:19,257 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/A is initiating minor compaction (all files) 2023-04-25 06:56:19,257 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/A in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,258 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a0a095c88d7e406c9b77aa61792a0fed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b781ebece9cb46e7a8db3369574ca0d3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/ad084ccca90844a793f4d32c26aa4797] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=83.0 K 2023-04-25 06:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,258 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:19,258 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/B is initiating minor compaction (all files) 2023-04-25 06:56:19,258 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/B in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,258 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a0a095c88d7e406c9b77aa61792a0fed, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682405776441 2023-04-25 06:56:19,258 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/8607862ed43b4a819e4b2dda3ea4b57e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/ed9fe92af43c4b348e0466bf9ca0cb6c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/0112556ad06e4f04a24d644d48f9dcbe] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=35.5 K 2023-04-25 06:56:19,259 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b781ebece9cb46e7a8db3369574ca0d3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682405776515 2023-04-25 06:56:19,259 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 8607862ed43b4a819e4b2dda3ea4b57e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682405776441 2023-04-25 06:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:19,259 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:19,259 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:19,259 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:19,259 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:19,259 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,259 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting ed9fe92af43c4b348e0466bf9ca0cb6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682405776515 2023-04-25 06:56:19,260 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting ad084ccca90844a793f4d32c26aa4797, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682405777644 2023-04-25 06:56:19,260 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 0112556ad06e4f04a24d644d48f9dcbe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682405777644 2023-04-25 06:56:19,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,261 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,262 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,265 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,269 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,269 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,269 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#B#compaction#502 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:19,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:19,279 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#A#compaction#503 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:19,299 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/fa67943f761a422eb713548e069b2060 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fa67943f761a422eb713548e069b2060 2023-04-25 06:56:19,303 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/A of e6358a6a069ce34b04034f9d45664d77 into fa67943f761a422eb713548e069b2060(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:19,303 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:19,303 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/A, priority=13, startTime=1682405779256; duration=0sec 2023-04-25 06:56:19,304 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:19,304 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 06:56:19,305 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 06:56:19,305 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/C is initiating minor compaction (all files) 2023-04-25 06:56:19,305 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/C in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,305 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/ea5230e047144784b4f9e4bae37a2a6b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/4b4ccb4429884d899dec3912c336bd7e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dc9140e428e548f7a0c2b9e18f9e81d7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3e2575af41274d90a381d23d11902c1d] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=47.1 K 2023-04-25 06:56:19,305 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting ea5230e047144784b4f9e4bae37a2a6b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682405775783 2023-04-25 06:56:19,306 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 4b4ccb4429884d899dec3912c336bd7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682405776441 2023-04-25 06:56:19,306 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting dc9140e428e548f7a0c2b9e18f9e81d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682405776515 2023-04-25 06:56:19,306 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 3e2575af41274d90a381d23d11902c1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682405777644 2023-04-25 06:56:19,314 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#C#compaction#504 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:19,369 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:19,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:19,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:19,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:19,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:19,372 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:19,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:19,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:19,372 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,373 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,374 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,374 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,374 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,374 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:19,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:19,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:19,376 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:56:19,376 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:19,376 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:19,376 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:19,376 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:19,376 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:19,376 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:19,392 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425412ebe9caf15414da6cad8b9e2bd9a75 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425412ebe9caf15414da6cad8b9e2bd9a75 2023-04-25 06:56:19,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:19,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:19,398 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=211, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/08bd17558bd440acac2d7a5a1e6a608d 2023-04-25 06:56:19,406 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/7dbfb36bae1947f3a8c3122efcb7a5e9 2023-04-25 06:56:19,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405839407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405839408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405839408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405839409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,414 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/5a7c8fb1d37c4c09a8e43a33a5b5f001 2023-04-25 06:56:19,417 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/08bd17558bd440acac2d7a5a1e6a608d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/08bd17558bd440acac2d7a5a1e6a608d 2023-04-25 06:56:19,420 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/08bd17558bd440acac2d7a5a1e6a608d, entries=150, sequenceid=211, filesize=25.3 K 2023-04-25 06:56:19,421 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/7dbfb36bae1947f3a8c3122efcb7a5e9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/7dbfb36bae1947f3a8c3122efcb7a5e9 2023-04-25 06:56:19,424 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/7dbfb36bae1947f3a8c3122efcb7a5e9, entries=150, sequenceid=211, filesize=11.8 K 2023-04-25 06:56:19,425 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/5a7c8fb1d37c4c09a8e43a33a5b5f001 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5a7c8fb1d37c4c09a8e43a33a5b5f001 2023-04-25 06:56:19,427 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5a7c8fb1d37c4c09a8e43a33a5b5f001, entries=150, sequenceid=211, filesize=11.8 K 2023-04-25 06:56:19,428 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e6358a6a069ce34b04034f9d45664d77 in 52ms, sequenceid=211, compaction requested=false 2023-04-25 06:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:19,430 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,430 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,431 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,431 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:19,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42085d34[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:19,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:19,432 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,433 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:19,434 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,434 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:19,435 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:19,437 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:19,437 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:19,437 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:19,437 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:19,437 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,443 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,443 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,443 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,445 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,448 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,448 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,448 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,448 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:19,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:19,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 06:56:19,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:19,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:19,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:19,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:19,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:19,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:19,528 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255dc7abe1be984e47aa4e02ac045a721b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255dc7abe1be984e47aa4e02ac045a721b 2023-04-25 06:56:19,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405839531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405839532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405839532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405839533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,548 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:19,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:19,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:19,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:19,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:19,551 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:19,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:19,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:19,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,557 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,559 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,560 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:19,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:19,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:19,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:19,561 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,561 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:19,562 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:19,562 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,563 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d75cac4[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:19,565 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:19,566 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,566 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:19,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:19,567 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:19,567 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,567 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:19,569 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:19,571 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:19,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:19,571 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:19,572 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:19,572 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,573 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,574 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,575 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,578 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,578 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,578 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,578 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:19,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405839637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405839638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405839639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405839639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,678 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:19,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:19,680 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/f4d1e308bb7f4a0a9a6d33aaf35a566f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/f4d1e308bb7f4a0a9a6d33aaf35a566f 2023-04-25 06:56:19,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:19,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:19,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:19,682 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:19,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:19,682 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:19,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,685 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:19,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,685 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/B of e6358a6a069ce34b04034f9d45664d77 into f4d1e308bb7f4a0a9a6d33aaf35a566f(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:19,685 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:19,685 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/B, priority=13, startTime=1682405779256; duration=0sec 2023-04-25 06:56:19,685 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:19,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:19,686 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,687 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:19,687 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:19,687 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:19,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:19,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:19,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:19,689 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4575fe8b[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:19,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:19,691 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,692 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:19,692 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,692 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:19,694 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:19,697 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:19,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:19,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:19,697 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:19,697 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,699 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:19,705 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,705 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,705 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,706 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,706 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,705 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,726 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/831c883207464bdaaf012a1cc0de32d0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/831c883207464bdaaf012a1cc0de32d0 2023-04-25 06:56:19,731 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/C of e6358a6a069ce34b04034f9d45664d77 into 831c883207464bdaaf012a1cc0de32d0(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:19,731 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:19,731 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/C, priority=12, startTime=1682405779257; duration=0sec 2023-04-25 06:56:19,731 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:19,806 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:19,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:19,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:19,809 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:19,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:19,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:19,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,810 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,810 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,811 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,811 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:19,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:19,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:19,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a581e03[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:19,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:19,816 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,817 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:19,817 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,817 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:19,819 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,819 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,819 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,819 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:19,821 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:19,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:19,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:19,821 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:19,821 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,822 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,823 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,823 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,824 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,829 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,829 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,829 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,829 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,829 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,829 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,829 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405839840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405839842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405839842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:19,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405839842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,929 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:19,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:19,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:19,933 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:19,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:19,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:19,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,934 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/845ba9f5cff8432bab6b3ced18cf19f0 2023-04-25 06:56:19,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,935 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:19,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:19,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:19,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:19,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:19,937 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,937 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:19,937 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:19,937 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:19,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:19,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:19,937 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:19,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,939 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,939 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a390f81[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:19,942 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:19,943 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,945 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:19,945 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,945 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:19,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/6633fecf6ddd45b0a77e37fd5b69100c 2023-04-25 06:56:19,946 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:19,949 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:19,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:19,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:19,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:19,949 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:19,951 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,951 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:19,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,951 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:19,952 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:19,957 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:19,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/5b51d93291b14009ab5b0d389b2d3589 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:19,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:19,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:19,957 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:19,957 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:19,958 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:19,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:19,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:19,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/845ba9f5cff8432bab6b3ced18cf19f0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/845ba9f5cff8432bab6b3ced18cf19f0 2023-04-25 06:56:19,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/845ba9f5cff8432bab6b3ced18cf19f0, entries=150, sequenceid=224, filesize=25.3 K 2023-04-25 06:56:19,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/6633fecf6ddd45b0a77e37fd5b69100c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6633fecf6ddd45b0a77e37fd5b69100c 2023-04-25 06:56:19,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6633fecf6ddd45b0a77e37fd5b69100c, entries=150, sequenceid=224, filesize=11.8 K 2023-04-25 06:56:19,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/5b51d93291b14009ab5b0d389b2d3589 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5b51d93291b14009ab5b0d389b2d3589 2023-04-25 06:56:19,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5b51d93291b14009ab5b0d389b2d3589, entries=150, sequenceid=224, filesize=11.8 K 2023-04-25 06:56:19,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e6358a6a069ce34b04034f9d45664d77 in 458ms, sequenceid=224, compaction requested=true 2023-04-25 06:56:19,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:19,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:19,974 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:19,974 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:19,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:19,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:19,975 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78064 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:19,975 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/A is initiating minor compaction (all files) 2023-04-25 06:56:19,975 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/A in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,975 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fa67943f761a422eb713548e069b2060, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/08bd17558bd440acac2d7a5a1e6a608d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/845ba9f5cff8432bab6b3ced18cf19f0] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=76.2 K 2023-04-25 06:56:19,975 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:19,975 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/B is initiating minor compaction (all files) 2023-04-25 06:56:19,975 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/B in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:19,975 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/f4d1e308bb7f4a0a9a6d33aaf35a566f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/7dbfb36bae1947f3a8c3122efcb7a5e9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6633fecf6ddd45b0a77e37fd5b69100c] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=35.6 K 2023-04-25 06:56:19,976 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting fa67943f761a422eb713548e069b2060, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682405777644 2023-04-25 06:56:19,976 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting f4d1e308bb7f4a0a9a6d33aaf35a566f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682405777644 2023-04-25 06:56:19,976 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 08bd17558bd440acac2d7a5a1e6a608d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682405778781 2023-04-25 06:56:19,976 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7dbfb36bae1947f3a8c3122efcb7a5e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682405778781 2023-04-25 06:56:19,977 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 845ba9f5cff8432bab6b3ced18cf19f0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682405779406 2023-04-25 06:56:19,977 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6633fecf6ddd45b0a77e37fd5b69100c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682405779406 2023-04-25 06:56:19,984 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#B#compaction#512 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:19,997 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#A#compaction#511 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:20,009 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/c3eb1538705d4a38a980a303894862e4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c3eb1538705d4a38a980a303894862e4 2023-04-25 06:56:20,013 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/B of e6358a6a069ce34b04034f9d45664d77 into c3eb1538705d4a38a980a303894862e4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:20,013 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:20,013 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/B, priority=13, startTime=1682405779974; duration=0sec 2023-04-25 06:56:20,013 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:20,014 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:20,015 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:20,015 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/C is initiating minor compaction (all files) 2023-04-25 06:56:20,015 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/C in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,015 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/831c883207464bdaaf012a1cc0de32d0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5a7c8fb1d37c4c09a8e43a33a5b5f001, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5b51d93291b14009ab5b0d389b2d3589] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=35.6 K 2023-04-25 06:56:20,015 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 831c883207464bdaaf012a1cc0de32d0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682405777644 2023-04-25 06:56:20,015 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5a7c8fb1d37c4c09a8e43a33a5b5f001, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682405778781 2023-04-25 06:56:20,016 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5b51d93291b14009ab5b0d389b2d3589, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682405779406 2023-04-25 06:56:20,022 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#C#compaction#513 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:20,036 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/a09d8a4a121841d8b516f286df7699dc as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a09d8a4a121841d8b516f286df7699dc 2023-04-25 06:56:20,041 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/C of e6358a6a069ce34b04034f9d45664d77 into a09d8a4a121841d8b516f286df7699dc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:20,042 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:20,042 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/C, priority=13, startTime=1682405779974; duration=0sec 2023-04-25 06:56:20,042 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:20,058 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:20,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:20,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:20,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:20,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:20,061 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:20,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:20,061 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:20,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,063 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,064 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,064 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,064 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:20,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:20,065 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,065 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:20,066 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:56:20,066 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:20,066 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:20,066 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:20,066 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:20,066 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:20,066 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:20,087 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d1d41c7a1f36431dac7b77e0128c2322 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d1d41c7a1f36431dac7b77e0128c2322 2023-04-25 06:56:20,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:20,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:20,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405840157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405840159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405840160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405840160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405840261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405840262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405840264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405840264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,428 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/fbaf317858ee431fb54e869c3335d79e as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fbaf317858ee431fb54e869c3335d79e 2023-04-25 06:56:20,433 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/A of e6358a6a069ce34b04034f9d45664d77 into fbaf317858ee431fb54e869c3335d79e(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:20,433 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:20,433 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/A, priority=13, startTime=1682405779974; duration=0sec 2023-04-25 06:56:20,433 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:20,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405840464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405840466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405840468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405840469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,494 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=247, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/6d65fa411e2d45b08a3f39c88ac0edd2 2023-04-25 06:56:20,508 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/771ef43f41164926a21624ce5e00ad99 2023-04-25 06:56:20,516 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/3018094a21bd44af94e1c7c8e960efa7 2023-04-25 06:56:20,519 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/6d65fa411e2d45b08a3f39c88ac0edd2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6d65fa411e2d45b08a3f39c88ac0edd2 2023-04-25 06:56:20,522 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6d65fa411e2d45b08a3f39c88ac0edd2, entries=150, sequenceid=247, filesize=25.3 K 2023-04-25 06:56:20,523 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/771ef43f41164926a21624ce5e00ad99 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/771ef43f41164926a21624ce5e00ad99 2023-04-25 06:56:20,526 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/771ef43f41164926a21624ce5e00ad99, entries=150, sequenceid=247, filesize=11.8 K 2023-04-25 06:56:20,527 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/3018094a21bd44af94e1c7c8e960efa7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3018094a21bd44af94e1c7c8e960efa7 2023-04-25 06:56:20,531 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3018094a21bd44af94e1c7c8e960efa7, entries=150, sequenceid=247, filesize=11.8 K 2023-04-25 06:56:20,532 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for e6358a6a069ce34b04034f9d45664d77 in 465ms, sequenceid=247, compaction requested=false 2023-04-25 06:56:20,532 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:20,532 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:20,533 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,533 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:20,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4939dbe8[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:20,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:20,536 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,537 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:20,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,537 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:20,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:20,537 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:20,538 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,538 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,538 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,538 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:20,542 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:20,542 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:20,542 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:20,542 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:20,542 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:20,545 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,545 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,545 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,546 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,548 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,550 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,550 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:20,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:20,551 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,551 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,551 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:20,551 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:20,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:20,651 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:20,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:20,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:20,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:20,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:20,654 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:20,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:20,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:20,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,655 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,656 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:20,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:56:20,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:20,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:20,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:20,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:20,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:20,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:20,658 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:20,658 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,658 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:20,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:20,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:20,659 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:20,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,662 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,663 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:20,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d2e5e2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:20,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:20,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,669 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:20,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,669 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:20,670 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:20,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,670 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:20,674 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:20,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:20,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:20,674 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:20,674 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:20,675 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,675 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,676 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,677 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,678 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,683 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:20,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:20,684 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,684 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:20,684 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:20,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:20,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:20,691 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252b6c289ee2fc4965a91b98987a5e8546 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252b6c289ee2fc4965a91b98987a5e8546 2023-04-25 06:56:20,696 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=263, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/12499782513442e9a1ee1841a81cfb78 2023-04-25 06:56:20,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/292b9eb770944813ae6c1d23aeb37442 2023-04-25 06:56:20,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405840714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405840765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405840770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405840770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405840771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,784 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:20,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:20,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:20,788 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:20,789 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,789 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:20,789 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:20,789 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,789 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,790 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,790 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,791 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:20,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:20,792 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,792 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:20,792 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:20,793 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:20,794 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,794 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,794 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d05f9d[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:20,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:20,797 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,799 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:20,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:20,799 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:20,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,799 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:20,800 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,800 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:20,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,801 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:20,803 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:20,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:20,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:20,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:20,803 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:20,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,804 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:20,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,807 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,809 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:20,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:20,809 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,810 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:20,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:20,810 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:20,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:20,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405840817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,910 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:20,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:20,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:20,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:20,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:20,914 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:20,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:20,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:20,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,914 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,916 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,916 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,917 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:20,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:20,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:20,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:20,918 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,918 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:20,918 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:20,918 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,918 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:20,919 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:20,920 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,920 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb71c21[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:20,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:20,922 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,923 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,923 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:20,923 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:20,923 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:20,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,923 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:20,924 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:20,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,924 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:20,927 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:20,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:20,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:20,927 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:20,927 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:20,929 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,929 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,929 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:20,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:20,930 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,931 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,933 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:20,933 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:20,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:20,933 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:20,933 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:20,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:20,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:20,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:20,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:20,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:20,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:20,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:20,934 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:20,934 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:20,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:21,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405841022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,034 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:21,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:21,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:21,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:21,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:21,037 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:21,037 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:21,037 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:21,037 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:21,038 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,038 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,039 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,039 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:21,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:21,039 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,039 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:21,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:21,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:21,045 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:21,045 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:21,045 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,046 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:21,046 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:21,046 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:21,047 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,047 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:21,048 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,048 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:21,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:21,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c566424[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:21,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:21,049 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,053 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:21,053 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,053 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:21,054 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:21,054 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:21,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:21,057 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:21,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:21,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:21,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:21,057 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:21,058 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,058 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,058 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:21,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:21,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:21,059 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,060 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:21,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:21,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:21,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:21,062 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:21,062 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:21,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:21,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/6332d342fe0f4418b2be944c178ab4e6 2023-04-25 06:56:21,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/12499782513442e9a1ee1841a81cfb78 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/12499782513442e9a1ee1841a81cfb78 2023-04-25 06:56:21,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/12499782513442e9a1ee1841a81cfb78, entries=200, sequenceid=263, filesize=32.3 K 2023-04-25 06:56:21,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/292b9eb770944813ae6c1d23aeb37442 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/292b9eb770944813ae6c1d23aeb37442 2023-04-25 06:56:21,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/292b9eb770944813ae6c1d23aeb37442, entries=150, sequenceid=263, filesize=11.9 K 2023-04-25 06:56:21,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/6332d342fe0f4418b2be944c178ab4e6 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6332d342fe0f4418b2be944c178ab4e6 2023-04-25 06:56:21,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6332d342fe0f4418b2be944c178ab4e6, entries=150, sequenceid=263, filesize=11.9 K 2023-04-25 06:56:21,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e6358a6a069ce34b04034f9d45664d77 in 473ms, sequenceid=263, compaction requested=true 2023-04-25 06:56:21,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:21,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:21,130 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:21,131 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:21,131 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:21,131 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:21,131 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85306 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:21,131 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/A is initiating minor compaction (all files) 2023-04-25 06:56:21,131 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36686 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:21,132 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/A in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,132 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/B is initiating minor compaction (all files) 2023-04-25 06:56:21,132 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fbaf317858ee431fb54e869c3335d79e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6d65fa411e2d45b08a3f39c88ac0edd2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/12499782513442e9a1ee1841a81cfb78] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=83.3 K 2023-04-25 06:56:21,132 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/B in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,132 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c3eb1538705d4a38a980a303894862e4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/771ef43f41164926a21624ce5e00ad99, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/292b9eb770944813ae6c1d23aeb37442] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=35.8 K 2023-04-25 06:56:21,132 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting fbaf317858ee431fb54e869c3335d79e, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682405779406 2023-04-25 06:56:21,133 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c3eb1538705d4a38a980a303894862e4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682405779406 2023-04-25 06:56:21,133 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 6d65fa411e2d45b08a3f39c88ac0edd2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1682405779529 2023-04-25 06:56:21,133 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 771ef43f41164926a21624ce5e00ad99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1682405779529 2023-04-25 06:56:21,133 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 12499782513442e9a1ee1841a81cfb78, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682405780155 2023-04-25 06:56:21,134 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 292b9eb770944813ae6c1d23aeb37442, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682405780155 2023-04-25 06:56:21,158 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#B#compaction#521 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:21,161 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#A#compaction#520 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:21,163 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:21,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:21,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:21,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:21,172 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:21,172 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:21,172 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:21,172 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:21,172 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,172 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,175 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:21,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:21,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,175 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:21,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:21,180 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:56:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:21,180 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:21,181 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:21,181 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:21,181 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:21,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:21,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:21,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405841284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405841285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405841286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405841287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405841327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405841388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405841389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405841389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405841391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,579 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/c53ecd35a1924ca7ad09a245840a9b32 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c53ecd35a1924ca7ad09a245840a9b32 2023-04-25 06:56:21,583 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/B of e6358a6a069ce34b04034f9d45664d77 into c53ecd35a1924ca7ad09a245840a9b32(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:21,583 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:21,583 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/B, priority=13, startTime=1682405781130; duration=0sec 2023-04-25 06:56:21,583 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:21,583 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 06:56:21,584 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36652 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 06:56:21,584 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/C is initiating minor compaction (all files) 2023-04-25 06:56:21,584 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/C in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,584 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a09d8a4a121841d8b516f286df7699dc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3018094a21bd44af94e1c7c8e960efa7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6332d342fe0f4418b2be944c178ab4e6] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=35.8 K 2023-04-25 06:56:21,584 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a09d8a4a121841d8b516f286df7699dc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682405779406 2023-04-25 06:56:21,585 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 3018094a21bd44af94e1c7c8e960efa7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1682405779529 2023-04-25 06:56:21,585 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6332d342fe0f4418b2be944c178ab4e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682405780155 2023-04-25 06:56:21,591 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#C#compaction#523 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405841592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405841592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405841593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405841594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,618 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042541dcb9975bb64cdb9c183411f531be80 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042541dcb9975bb64cdb9c183411f531be80 2023-04-25 06:56:21,625 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/045a6e1c92094a87a448f01227da93ca 2023-04-25 06:56:21,634 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e9756781601641d3a9dea13444e982f2 2023-04-25 06:56:21,655 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/01e44e0da5ad4a4496cfff0abc550335 2023-04-25 06:56:21,658 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/045a6e1c92094a87a448f01227da93ca as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/045a6e1c92094a87a448f01227da93ca 2023-04-25 06:56:21,661 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/045a6e1c92094a87a448f01227da93ca, entries=150, sequenceid=280, filesize=25.4 K 2023-04-25 06:56:21,662 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e9756781601641d3a9dea13444e982f2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9756781601641d3a9dea13444e982f2 2023-04-25 06:56:21,666 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9756781601641d3a9dea13444e982f2, entries=150, sequenceid=280, filesize=11.9 K 2023-04-25 06:56:21,666 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/01e44e0da5ad4a4496cfff0abc550335 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/01e44e0da5ad4a4496cfff0abc550335 2023-04-25 06:56:21,677 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/01e44e0da5ad4a4496cfff0abc550335, entries=150, sequenceid=280, filesize=11.9 K 2023-04-25 06:56:21,678 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e6358a6a069ce34b04034f9d45664d77 in 498ms, sequenceid=280, compaction requested=false 2023-04-25 06:56:21,678 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:21,678 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:21,682 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:21,682 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,682 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:21,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@189f69b9[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:21,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:21,684 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:21,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:21,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:21,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:21,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:21,690 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:21,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:21,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:21,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:21,690 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:21,693 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,693 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,693 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:21,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:21,694 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,696 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,698 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:21,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:21,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:21,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:21,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:21,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:21,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,698 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:21,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,699 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:21,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:21,699 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,699 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:21,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:21,799 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:21,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:21,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:21,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:21,802 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:21,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:21,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:21,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:21,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,802 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,803 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,803 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:21,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:21,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,804 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:21,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:21,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:21,805 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:21,805 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:21,805 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:21,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:21,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405841894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405841896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405841897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405841898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405841898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,002 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/1c3c59de0d7c4c48848626665191df31 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1c3c59de0d7c4c48848626665191df31 2023-04-25 06:56:22,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405842002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,006 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/C of e6358a6a069ce34b04034f9d45664d77 into 1c3c59de0d7c4c48848626665191df31(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:22,006 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:22,006 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/C, priority=13, startTime=1682405781131; duration=0sec 2023-04-25 06:56:22,006 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:22,011 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/a472be23ee8449eaa8381b246e657a84 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a472be23ee8449eaa8381b246e657a84 2023-04-25 06:56:22,015 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/A of e6358a6a069ce34b04034f9d45664d77 into a472be23ee8449eaa8381b246e657a84(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:22,015 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:22,015 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/A, priority=13, startTime=1682405781130; duration=0sec 2023-04-25 06:56:22,015 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405842205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,224 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042575bda3f3538d49dda4ac5ad895b935ce to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042575bda3f3538d49dda4ac5ad895b935ce 2023-04-25 06:56:22,230 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=296, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/7b6261e5579c495ebab226081f962b9b 2023-04-25 06:56:22,243 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e9b92c498561406ea9e0db59cd2693ff 2023-04-25 06:56:22,252 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/727ef22ad0e0443ab2a184d2b268a3f4 2023-04-25 06:56:22,256 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/7b6261e5579c495ebab226081f962b9b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7b6261e5579c495ebab226081f962b9b 2023-04-25 06:56:22,260 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7b6261e5579c495ebab226081f962b9b, entries=150, sequenceid=296, filesize=25.4 K 2023-04-25 06:56:22,261 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e9b92c498561406ea9e0db59cd2693ff as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9b92c498561406ea9e0db59cd2693ff 2023-04-25 06:56:22,268 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9b92c498561406ea9e0db59cd2693ff, entries=150, sequenceid=296, filesize=11.9 K 2023-04-25 06:56:22,269 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/727ef22ad0e0443ab2a184d2b268a3f4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/727ef22ad0e0443ab2a184d2b268a3f4 2023-04-25 06:56:22,272 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/727ef22ad0e0443ab2a184d2b268a3f4, entries=150, sequenceid=296, filesize=11.9 K 2023-04-25 06:56:22,272 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e6358a6a069ce34b04034f9d45664d77 in 467ms, sequenceid=296, compaction requested=true 2023-04-25 06:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:22,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,274 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1338f180[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:22,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:22,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:22,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:22,278 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:22,279 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,279 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:22,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:22,283 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:22,283 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:22,283 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:22,283 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:22,283 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:22,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:22,284 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,285 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:22,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:22,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:22,285 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:22,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:22,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,289 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:22,289 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:22,289 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:22,290 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:22,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:22,390 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:22,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:22,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:22,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:22,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:22,393 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:22,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:22,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:22,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:22,393 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:22,396 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:22,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:22,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,396 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:22,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:22,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:22,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:22,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:22,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:22,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:22,398 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 06:56:22,398 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:22,398 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:22,398 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:22,398 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:22,398 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:22,398 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:22,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:22,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:22,413 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042553cdbb225f3e4414a2cd0e916179139f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042553cdbb225f3e4414a2cd0e916179139f 2023-04-25 06:56:22,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405842414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405842415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405842416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405842417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,421 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/9ee112f7e4a44cc399d29044f2945366 2023-04-25 06:56:22,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405842509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405842518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405842518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405842520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39894 deadline: 1682405842522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405842721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405842722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405842722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:22,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405842724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:22,830 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/de3607b53639429da2633ffdc6e412d4 2023-04-25 06:56:23,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405843013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1682405843025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39940 deadline: 1682405843027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39930 deadline: 1682405843027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405843030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,244 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/7e28931c5e5248ecb709f6690bc5a8d4 2023-04-25 06:56:23,248 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/9ee112f7e4a44cc399d29044f2945366 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9ee112f7e4a44cc399d29044f2945366 2023-04-25 06:56:23,254 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9ee112f7e4a44cc399d29044f2945366, entries=150, sequenceid=315, filesize=25.4 K 2023-04-25 06:56:23,254 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/de3607b53639429da2633ffdc6e412d4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/de3607b53639429da2633ffdc6e412d4 2023-04-25 06:56:23,258 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/de3607b53639429da2633ffdc6e412d4, entries=150, sequenceid=315, filesize=11.9 K 2023-04-25 06:56:23,258 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/7e28931c5e5248ecb709f6690bc5a8d4 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e28931c5e5248ecb709f6690bc5a8d4 2023-04-25 06:56:23,261 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e28931c5e5248ecb709f6690bc5a8d4, entries=150, sequenceid=315, filesize=11.9 K 2023-04-25 06:56:23,262 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e6358a6a069ce34b04034f9d45664d77 in 864ms, sequenceid=315, compaction requested=true 2023-04-25 06:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:23,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:23,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:23,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:23,263 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:23,264 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,264 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,265 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:23,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fda1d2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:23,266 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:23,266 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,267 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:23,267 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:23,269 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:23,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:23,271 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:23,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:23,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:23,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:23,271 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:23,275 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,276 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,276 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:23,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,278 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:23,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:23,280 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:23,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:23,280 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:23,280 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:23,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:23,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:23,381 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:23,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:23,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:23,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:23,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:23,384 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:23,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:23,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:23,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:23,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,386 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:23,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:23,386 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,386 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,386 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:23,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:23,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:23,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:23,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:23,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:23,389 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 06:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:23,401 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042580bbf71b753a4896b7e0d2c2c716471f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042580bbf71b753a4896b7e0d2c2c716471f 2023-04-25 06:56:23,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:23,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:23,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405843544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405843546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405843546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405843549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405843650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405843650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405843650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405843653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,806 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=330, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/f976d6e59ab84365b7ed325e85d32e17 2023-04-25 06:56:23,815 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e3fe11a539344c849f2bffe9e995c2aa 2023-04-25 06:56:23,824 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/9aabf9c3187b4a8abb16ce8eae8b4b49 2023-04-25 06:56:23,828 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/f976d6e59ab84365b7ed325e85d32e17 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/f976d6e59ab84365b7ed325e85d32e17 2023-04-25 06:56:23,831 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/f976d6e59ab84365b7ed325e85d32e17, entries=150, sequenceid=330, filesize=25.4 K 2023-04-25 06:56:23,832 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e3fe11a539344c849f2bffe9e995c2aa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e3fe11a539344c849f2bffe9e995c2aa 2023-04-25 06:56:23,836 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e3fe11a539344c849f2bffe9e995c2aa, entries=150, sequenceid=330, filesize=11.9 K 2023-04-25 06:56:23,836 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/9aabf9c3187b4a8abb16ce8eae8b4b49 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/9aabf9c3187b4a8abb16ce8eae8b4b49 2023-04-25 06:56:23,839 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/9aabf9c3187b4a8abb16ce8eae8b4b49, entries=150, sequenceid=330, filesize=11.9 K 2023-04-25 06:56:23,840 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e6358a6a069ce34b04034f9d45664d77 in 451ms, sequenceid=330, compaction requested=true 2023-04-25 06:56:23,840 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:23,840 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:23,842 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1592ad[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:23,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:23,846 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,854 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,854 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:23,854 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:23,854 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:23,854 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,854 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:23,855 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:23,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:23,857 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:23,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:23,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:23,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:23,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:23,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:23,859 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:23,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:23,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:23,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:23,859 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:23,861 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,862 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,864 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,866 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:23,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:23,869 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:23,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:23,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:23,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:23,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,869 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:23,869 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:23,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:23,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405843873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405843875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405843876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405843876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,970 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:23,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:23,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:23,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:23,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:23,973 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:23,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:23,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:23,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:23,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,975 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:23,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:23,976 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,976 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:23,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:23,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:23,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:23,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:23,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:23,977 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:23,978 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:23,978 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:23,978 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:23,978 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:23,979 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,980 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:23,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405843978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:23,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c24f257[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:23,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:23,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405843980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,982 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405843981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405843981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,983 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:23,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:23,984 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:23,984 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,984 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:23,986 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:23,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,986 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:23,989 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:23,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:23,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:23,989 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:23,989 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:23,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:23,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,992 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:23,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,992 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,994 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:23,996 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:23,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:23,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:23,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:23,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:23,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:23,996 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:23,997 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:23,997 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:23,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:23,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:23,997 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:23,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405844018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,097 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:24,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:24,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:24,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:24,101 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:24,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:24,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:24,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:24,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,101 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,102 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:24,102 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:24,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,103 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:24,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:24,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:24,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,104 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:24,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:24,105 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:24,107 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,107 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:24,107 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,107 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c4b8a2[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:24,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:24,109 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,110 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:24,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,110 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:24,112 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:24,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:24,112 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:24,115 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:24,115 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:24,115 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:24,115 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:24,115 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:24,117 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,117 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:24,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,117 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:24,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,119 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,121 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:24,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,124 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:24,124 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:24,124 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:24,125 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:24,125 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:24,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:24,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:24,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405844181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405844184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405844184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405844185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,225 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:24,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:24,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:24,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:24,228 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:24,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:24,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:24,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:24,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,229 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,230 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:24,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:24,230 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,231 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:24,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:24,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:24,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:24,232 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,232 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:24,232 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:24,233 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:24,234 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,234 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:24,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,235 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:24,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:24,236 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45da7f59[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:24,237 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:24,238 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,238 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:24,242 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:24,242 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:24,243 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:24,245 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:24,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:24,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:24,245 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:24,245 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:24,247 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,247 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,247 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:24,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:24,247 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:24,249 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:24,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:24,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:24,251 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:24,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:24,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:24,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:24,251 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,252 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,252 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,252 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,252 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:24,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:24,252 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:24,252 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:24,252 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,271 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425cefad2309fb6456b8c15b7a53c0430f8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cefad2309fb6456b8c15b7a53c0430f8 2023-04-25 06:56:24,277 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=350, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/e8fc26e388c64ff8838c3c6abe638d14 2023-04-25 06:56:24,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/b9b9cb6b1a5b44228ddd7d34c94e8caa 2023-04-25 06:56:24,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/b0cfa60029514a5b896212fc7fb2aeb5 2023-04-25 06:56:24,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/e8fc26e388c64ff8838c3c6abe638d14 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/e8fc26e388c64ff8838c3c6abe638d14 2023-04-25 06:56:24,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/e8fc26e388c64ff8838c3c6abe638d14, entries=200, sequenceid=350, filesize=32.3 K 2023-04-25 06:56:24,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/b9b9cb6b1a5b44228ddd7d34c94e8caa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b9b9cb6b1a5b44228ddd7d34c94e8caa 2023-04-25 06:56:24,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b9b9cb6b1a5b44228ddd7d34c94e8caa, entries=150, sequenceid=350, filesize=11.9 K 2023-04-25 06:56:24,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/b0cfa60029514a5b896212fc7fb2aeb5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/b0cfa60029514a5b896212fc7fb2aeb5 2023-04-25 06:56:24,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/b0cfa60029514a5b896212fc7fb2aeb5, entries=150, sequenceid=350, filesize=11.9 K 2023-04-25 06:56:24,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e6358a6a069ce34b04034f9d45664d77 in 471ms, sequenceid=350, compaction requested=true 2023-04-25 06:56:24,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:24,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:24,328 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:56:24,329 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:56:24,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:24,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:24,330 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73646 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:56:24,330 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/B is initiating minor compaction (all files) 2023-04-25 06:56:24,330 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/B in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,330 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163846 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:56:24,330 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c53ecd35a1924ca7ad09a245840a9b32, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9756781601641d3a9dea13444e982f2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9b92c498561406ea9e0db59cd2693ff, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/de3607b53639429da2633ffdc6e412d4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e3fe11a539344c849f2bffe9e995c2aa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b9b9cb6b1a5b44228ddd7d34c94e8caa] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=71.9 K 2023-04-25 06:56:24,330 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/A is initiating minor compaction (all files) 2023-04-25 06:56:24,331 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/A in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,331 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a472be23ee8449eaa8381b246e657a84, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/045a6e1c92094a87a448f01227da93ca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7b6261e5579c495ebab226081f962b9b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9ee112f7e4a44cc399d29044f2945366, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/f976d6e59ab84365b7ed325e85d32e17, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/e8fc26e388c64ff8838c3c6abe638d14] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=160.0 K 2023-04-25 06:56:24,331 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting c53ecd35a1924ca7ad09a245840a9b32, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682405780155 2023-04-25 06:56:24,331 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting a472be23ee8449eaa8381b246e657a84, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682405780155 2023-04-25 06:56:24,331 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e9756781601641d3a9dea13444e982f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682405780673 2023-04-25 06:56:24,331 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 045a6e1c92094a87a448f01227da93ca, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682405780673 2023-04-25 06:56:24,332 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e9b92c498561406ea9e0db59cd2693ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682405781284 2023-04-25 06:56:24,332 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 7b6261e5579c495ebab226081f962b9b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682405781284 2023-04-25 06:56:24,332 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting de3607b53639429da2633ffdc6e412d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682405781868 2023-04-25 06:56:24,332 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 9ee112f7e4a44cc399d29044f2945366, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682405781868 2023-04-25 06:56:24,332 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting e3fe11a539344c849f2bffe9e995c2aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682405782398 2023-04-25 06:56:24,333 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting f976d6e59ab84365b7ed325e85d32e17, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682405782398 2023-04-25 06:56:24,333 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b9b9cb6b1a5b44228ddd7d34c94e8caa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682405783544 2023-04-25 06:56:24,333 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting e8fc26e388c64ff8838c3c6abe638d14, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682405783544 2023-04-25 06:56:24,343 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#B#compaction#538 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:24,346 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#A#compaction#539 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:24,352 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:24,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:24,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:24,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:24,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:24,363 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:24,363 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:24,363 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:24,363 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:24,363 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,363 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,365 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:24,365 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:24,366 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,366 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:24,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:24,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:24,376 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:24,376 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:24,376 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:24,380 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:24,380 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:24,381 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:24,381 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:24,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:24,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:24,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405844506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405844507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405844508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405844510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405844611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405844611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405844612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405844614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,761 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/fecd79c94ed5471bb2892cf976608f20 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/fecd79c94ed5471bb2892cf976608f20 2023-04-25 06:56:24,765 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/B of e6358a6a069ce34b04034f9d45664d77 into fecd79c94ed5471bb2892cf976608f20(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:24,765 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:24,765 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/B, priority=10, startTime=1682405784328; duration=0sec 2023-04-25 06:56:24,765 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:24,765 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:56:24,767 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73612 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:56:24,767 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/C is initiating minor compaction (all files) 2023-04-25 06:56:24,767 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/C in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:24,767 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1c3c59de0d7c4c48848626665191df31, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/01e44e0da5ad4a4496cfff0abc550335, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/727ef22ad0e0443ab2a184d2b268a3f4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e28931c5e5248ecb709f6690bc5a8d4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/9aabf9c3187b4a8abb16ce8eae8b4b49, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/b0cfa60029514a5b896212fc7fb2aeb5] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=71.9 K 2023-04-25 06:56:24,767 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1c3c59de0d7c4c48848626665191df31, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682405780155 2023-04-25 06:56:24,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 01e44e0da5ad4a4496cfff0abc550335, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682405780673 2023-04-25 06:56:24,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 727ef22ad0e0443ab2a184d2b268a3f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682405781284 2023-04-25 06:56:24,768 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 7e28931c5e5248ecb709f6690bc5a8d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682405781868 2023-04-25 06:56:24,769 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 9aabf9c3187b4a8abb16ce8eae8b4b49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682405782398 2023-04-25 06:56:24,769 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting b0cfa60029514a5b896212fc7fb2aeb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682405783544 2023-04-25 06:56:24,779 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#C#compaction#541 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:24,809 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042504925987441c495dbc7f151056a6b751 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042504925987441c495dbc7f151056a6b751 2023-04-25 06:56:24,813 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/98f774e507544397a989e5b86dc67ec1 2023-04-25 06:56:24,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405844815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405844816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405844816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:24,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:24,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405844820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405845120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405845120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405845122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405845124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,188 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/a168d58afd8a4a57893ad47bc5bd67dd as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a168d58afd8a4a57893ad47bc5bd67dd 2023-04-25 06:56:25,192 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/C of e6358a6a069ce34b04034f9d45664d77 into a168d58afd8a4a57893ad47bc5bd67dd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:25,192 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:25,192 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/C, priority=10, startTime=1682405784329; duration=0sec 2023-04-25 06:56:25,192 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:25,203 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/67cb8ffa91214b428843bf994d771828 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/67cb8ffa91214b428843bf994d771828 2023-04-25 06:56:25,207 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/A of e6358a6a069ce34b04034f9d45664d77 into 67cb8ffa91214b428843bf994d771828(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:25,207 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:25,207 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/A, priority=10, startTime=1682405784328; duration=0sec 2023-04-25 06:56:25,207 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:25,222 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/5bcc87e1dff34947901f766aaa81043c 2023-04-25 06:56:25,233 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/aed681f3758e440caaceba9a2c929c30 2023-04-25 06:56:25,238 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/98f774e507544397a989e5b86dc67ec1 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/98f774e507544397a989e5b86dc67ec1 2023-04-25 06:56:25,241 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/98f774e507544397a989e5b86dc67ec1, entries=150, sequenceid=362, filesize=25.4 K 2023-04-25 06:56:25,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/5bcc87e1dff34947901f766aaa81043c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/5bcc87e1dff34947901f766aaa81043c 2023-04-25 06:56:25,246 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/5bcc87e1dff34947901f766aaa81043c, entries=150, sequenceid=362, filesize=11.9 K 2023-04-25 06:56:25,247 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/aed681f3758e440caaceba9a2c929c30 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/aed681f3758e440caaceba9a2c929c30 2023-04-25 06:56:25,254 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/aed681f3758e440caaceba9a2c929c30, entries=150, sequenceid=362, filesize=11.9 K 2023-04-25 06:56:25,255 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e6358a6a069ce34b04034f9d45664d77 in 874ms, sequenceid=362, compaction requested=false 2023-04-25 06:56:25,255 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:25,255 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:25,257 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,257 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76691db5[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:25,259 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:25,259 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,260 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:25,261 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,261 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:25,262 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:25,265 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:25,265 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:25,265 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:25,266 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:25,266 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:25,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:25,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,269 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:25,269 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:25,270 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:25,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:25,273 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:25,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:25,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:25,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:25,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:25,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:25,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:25,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:25,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:25,274 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,274 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:25,274 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:25,374 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:25,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:25,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:25,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:25,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:25,384 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:25,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:25,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:25,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:25,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,384 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:25,385 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:25,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:25,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:25,386 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,386 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:25,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:25,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:25,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:25,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:25,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:25,388 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 06:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:25,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:25,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:25,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405845633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405845634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405845635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405845635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405845736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405845737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405845738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405845739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,807 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304259171e579b277478d92f1992d278e58e4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259171e579b277478d92f1992d278e58e4 2023-04-25 06:56:25,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405845939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405845940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405845942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:25,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:25,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405845942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405846036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,039 DEBUG [Thread-3618] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:56:26,217 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/b3e502ea9650467aa144e7749c581784 2023-04-25 06:56:26,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405846244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405846246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405846247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405846248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,627 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/1dd96b3329b94094b2932c943273ac8c 2023-04-25 06:56:26,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405846747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405846749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405846750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:26,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:26,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405846751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,037 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/5c3ad48a106749fe8f816d8ab85a479d 2023-04-25 06:56:27,040 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/b3e502ea9650467aa144e7749c581784 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b3e502ea9650467aa144e7749c581784 2023-04-25 06:56:27,046 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b3e502ea9650467aa144e7749c581784, entries=150, sequenceid=385, filesize=25.4 K 2023-04-25 06:56:27,047 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/1dd96b3329b94094b2932c943273ac8c as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/1dd96b3329b94094b2932c943273ac8c 2023-04-25 06:56:27,051 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/1dd96b3329b94094b2932c943273ac8c, entries=150, sequenceid=385, filesize=11.9 K 2023-04-25 06:56:27,051 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/5c3ad48a106749fe8f816d8ab85a479d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5c3ad48a106749fe8f816d8ab85a479d 2023-04-25 06:56:27,057 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5c3ad48a106749fe8f816d8ab85a479d, entries=150, sequenceid=385, filesize=11.9 K 2023-04-25 06:56:27,058 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e6358a6a069ce34b04034f9d45664d77 in 1671ms, sequenceid=385, compaction requested=true 2023-04-25 06:56:27,058 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:27,058 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,060 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:27,060 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:27,061 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,062 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,062 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:27,064 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:27,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d84ff65[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:27,066 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,068 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:27,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:27,072 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:27,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:27,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:27,072 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:27,072 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:27,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,074 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,076 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:27,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,078 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,079 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:27,079 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:27,079 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:27,179 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:27,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:27,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:27,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:27,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:27,183 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:27,183 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,183 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:27,183 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:27,183 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,184 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,186 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:27,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:27,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:27,187 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:27,187 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,187 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:27,187 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:27,187 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:27,187 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,187 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:27,188 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 06:56:27,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:27,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:27,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:27,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:27,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:27,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:27,209 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ec3341237109439ea2393543ece8bd89 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ec3341237109439ea2393543ece8bd89 2023-04-25 06:56:27,218 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=397, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/b8147ccbfd1248d9b76737fdf5e63ae9 2023-04-25 06:56:27,228 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/23ab25bc46224497a9d7b061a763726a 2023-04-25 06:56:27,238 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/893a5ba0e30f46198ccbf8c7a63d0fc3 2023-04-25 06:56:27,242 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/b8147ccbfd1248d9b76737fdf5e63ae9 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b8147ccbfd1248d9b76737fdf5e63ae9 2023-04-25 06:56:27,258 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b8147ccbfd1248d9b76737fdf5e63ae9, entries=150, sequenceid=397, filesize=25.4 K 2023-04-25 06:56:27,259 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/23ab25bc46224497a9d7b061a763726a as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/23ab25bc46224497a9d7b061a763726a 2023-04-25 06:56:27,262 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/23ab25bc46224497a9d7b061a763726a, entries=150, sequenceid=397, filesize=11.9 K 2023-04-25 06:56:27,263 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/893a5ba0e30f46198ccbf8c7a63d0fc3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/893a5ba0e30f46198ccbf8c7a63d0fc3 2023-04-25 06:56:27,266 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/893a5ba0e30f46198ccbf8c7a63d0fc3, entries=150, sequenceid=397, filesize=11.9 K 2023-04-25 06:56:27,267 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for e6358a6a069ce34b04034f9d45664d77 in 78ms, sequenceid=397, compaction requested=true 2023-04-25 06:56:27,267 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:27,267 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:27,268 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:27,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@208843df[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:27,271 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,272 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:27,272 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:27,272 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:27,273 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:27,276 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:27,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:27,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:27,276 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:27,276 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:27,279 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,279 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,279 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,280 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,281 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,286 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:27,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:27,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:27,286 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:27,287 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:27,286 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,287 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,387 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:27,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:27,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:27,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:27,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:27,391 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:27,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:27,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:27,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,391 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,394 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,394 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,395 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:27,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:27,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:27,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:27,397 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:27,398 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,399 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:27,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bec020[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:27,401 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:27,401 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,402 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:27,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,402 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:27,404 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:27,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:27,407 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:27,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:27,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:27,407 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:27,407 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:27,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,409 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,409 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,410 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,410 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:27,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,412 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,413 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:27,412 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,413 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,413 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,413 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,413 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:27,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:27,513 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:27,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:27,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:27,517 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:27,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:27,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:27,517 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,518 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,519 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,519 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:27,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:27,520 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,521 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:27,521 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:27,521 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:27,523 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,523 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@884e7f7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:27,525 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:27,525 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:27,526 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,526 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:27,527 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:27,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:27,530 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:27,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:27,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:27,530 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:27,530 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:27,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,532 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,533 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,534 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,540 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,540 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:27,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:27,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,541 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:27,541 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:27,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,641 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:27,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:27,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:27,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:27,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:27,645 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:27,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:27,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:27,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,645 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,646 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,647 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:27,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:27,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:27,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:27,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:27,649 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:27,650 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,651 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2714e55[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:27,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:27,653 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,654 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:27,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:27,654 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:27,654 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:27,656 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:27,656 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,656 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,656 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:27,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:27,660 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:27,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:27,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:27,660 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:27,660 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:27,661 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,661 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:27,662 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:27,662 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,664 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,666 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,666 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,666 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:27,666 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:27,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:27,666 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:27,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:27,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,666 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:27,667 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:27,666 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:27,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:27,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:27,667 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,767 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:27,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:27,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:27,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:27,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:27,771 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:27,771 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,771 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:27,771 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:27,771 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,772 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,774 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:27,776 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:27,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:27,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:27,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:27,790 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:27,790 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:27,790 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:27,790 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:27,790 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-25 06:56:27,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:27,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:27,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:27,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:27,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:27,799 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:27,831 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425590b2333b8df4eabb6d66fa889231a95 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425590b2333b8df4eabb6d66fa889231a95 2023-04-25 06:56:27,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:27,838 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:27,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405787852, max timestamp=1682405787859Num uniques 859; 2023-04-25 06:56:27,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405787852, max timestamp=1682405787859Num uniques 859; 2023-04-25 06:56:27,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:56:27,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682405787852, max timestamp=1682405787859Num uniques 859; 2023-04-25 06:56:27,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:56:27,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 06:56:27,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:27,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405847878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405847880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:27,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682405787864, max timestamp=1682405787881Num uniques 1118; 2023-04-25 06:56:27,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-25 06:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682405787864, max timestamp=1682405787881Num uniques 1118; 2023-04-25 06:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682405787864, max timestamp=1682405787881Num uniques 1118; 2023-04-25 06:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-25 06:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-25 06:56:27,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682405787911, max timestamp=1682405787924Num uniques 1127; 2023-04-25 06:56:27,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-25 06:56:27,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682405787911, max timestamp=1682405787924Num uniques 1127; 2023-04-25 06:56:27,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-25 06:56:27,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682405787911, max timestamp=1682405787924Num uniques 1077; 2023-04-25 06:56:27,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-25 06:56:27,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682405787949, max timestamp=1682405787962Num uniques 1036; 2023-04-25 06:56:27,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 06:56:27,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682405787949, max timestamp=1682405787962Num uniques 1036; 2023-04-25 06:56:27,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 06:56:27,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682405787949, max timestamp=1682405787962Num uniques 1036; 2023-04-25 06:56:27,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 06:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682405787971, max timestamp=1682405787998Num uniques 1045; 2023-04-25 06:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682405787971, max timestamp=1682405787998Num uniques 1045; 2023-04-25 06:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-25 06:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682405787971, max timestamp=1682405787998Num uniques 1045; 2023-04-25 06:56:28,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-25 06:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-25 06:56:28,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405848021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682405787994, max timestamp=1682405788029Num uniques 1104; 2023-04-25 06:56:28,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682405787994, max timestamp=1682405788021Num uniques 1104; 2023-04-25 06:56:28,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-25 06:56:28,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682405787994, max timestamp=1682405788021Num uniques 1104; 2023-04-25 06:56:28,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-25 06:56:28,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-25 06:56:28,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682405788059, max timestamp=1682405788068Num uniques 1013; 2023-04-25 06:56:28,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-25 06:56:28,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682405788059, max timestamp=1682405788068Num uniques 1013; 2023-04-25 06:56:28,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-25 06:56:28,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682405788059, max timestamp=1682405788068Num uniques 1013; 2023-04-25 06:56:28,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-25 06:56:28,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405848091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682405788075, max timestamp=1682405788092Num uniques 1072; 2023-04-25 06:56:28,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-25 06:56:28,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682405788075, max timestamp=1682405788092Num uniques 1072; 2023-04-25 06:56:28,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-25 06:56:28,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682405788075, max timestamp=1682405788092Num uniques 1072; 2023-04-25 06:56:28,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-25 06:56:28,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405848133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682405788127, max timestamp=1682405788136Num uniques 1131; 2023-04-25 06:56:28,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682405788127, max timestamp=1682405788136Num uniques 1131; 2023-04-25 06:56:28,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682405788127, max timestamp=1682405788136Num uniques 1131; 2023-04-25 06:56:28,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-25 06:56:28,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-25 06:56:28,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-25 06:56:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682405788165, max timestamp=1682405788178Num uniques 1040; 2023-04-25 06:56:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-25 06:56:28,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682405788165, max timestamp=1682405788178Num uniques 1040; 2023-04-25 06:56:28,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-25 06:56:28,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682405788165, max timestamp=1682405788178Num uniques 1040; 2023-04-25 06:56:28,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-25 06:56:28,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682405788207, max timestamp=1682405788225Num uniques 1099; 2023-04-25 06:56:28,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682405788207, max timestamp=1682405788225Num uniques 1099; 2023-04-25 06:56:28,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-25 06:56:28,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682405788207, max timestamp=1682405788225Num uniques 1099; 2023-04-25 06:56:28,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-25 06:56:28,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-25 06:56:28,238 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=407, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/afab958b0887452ab885dafd2f3e4ef7 2023-04-25 06:56:28,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682405788238, max timestamp=1682405788251Num uniques 1058; 2023-04-25 06:56:28,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-25 06:56:28,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682405788238, max timestamp=1682405788251Num uniques 1058; 2023-04-25 06:56:28,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-25 06:56:28,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1682405788238, max timestamp=1682405788251Num uniques 1058; 2023-04-25 06:56:28,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-25 06:56:28,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682405788274, max timestamp=1682405788282Num uniques 1067; 2023-04-25 06:56:28,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-25 06:56:28,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682405788274, max timestamp=1682405788282Num uniques 1067; 2023-04-25 06:56:28,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-25 06:56:28,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682405788274, max timestamp=1682405788282Num uniques 1067; 2023-04-25 06:56:28,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-25 06:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682405788325, max timestamp=1682405788331Num uniques 1076; 2023-04-25 06:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-25 06:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682405788325, max timestamp=1682405788331Num uniques 1076; 2023-04-25 06:56:28,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-25 06:56:28,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682405788325, max timestamp=1682405788331Num uniques 1076; 2023-04-25 06:56:28,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-25 06:56:28,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1682405788345, max timestamp=1682405788352Num uniques 1035; 2023-04-25 06:56:28,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-25 06:56:28,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1682405788345, max timestamp=1682405788352Num uniques 1035; 2023-04-25 06:56:28,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-25 06:56:28,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1682405788345, max timestamp=1682405788352Num uniques 1035; 2023-04-25 06:56:28,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-25 06:56:28,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405848382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405848383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682405788379, max timestamp=1682405788385Num uniques 1144; 2023-04-25 06:56:28,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-25 06:56:28,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682405788379, max timestamp=1682405788385Num uniques 1144; 2023-04-25 06:56:28,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-25 06:56:28,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682405788379, max timestamp=1682405788385Num uniques 1144; 2023-04-25 06:56:28,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-25 06:56:28,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682405788395, max timestamp=1682405788423Num uniques 1053; 2023-04-25 06:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-25 06:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682405788395, max timestamp=1682405788423Num uniques 1053; 2023-04-25 06:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-25 06:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682405788395, max timestamp=1682405788423Num uniques 1053; 2023-04-25 06:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-25 06:56:28,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682405788449, max timestamp=1682405788456Num uniques 1012; 2023-04-25 06:56:28,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-25 06:56:28,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682405788449, max timestamp=1682405788456Num uniques 1012; 2023-04-25 06:56:28,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682405788449, max timestamp=1682405788456Num uniques 1012; 2023-04-25 06:56:28,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-25 06:56:28,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-25 06:56:28,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682405788489, max timestamp=1682405788495Num uniques 1071; 2023-04-25 06:56:28,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-25 06:56:28,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682405788489, max timestamp=1682405788495Num uniques 1071; 2023-04-25 06:56:28,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-25 06:56:28,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682405788489, max timestamp=1682405788495Num uniques 1071; 2023-04-25 06:56:28,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-25 06:56:28,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682405788509, max timestamp=1682405788519Num uniques 1030; 2023-04-25 06:56:28,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-25 06:56:28,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682405788509, max timestamp=1682405788519Num uniques 1030; 2023-04-25 06:56:28,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682405788509, max timestamp=1682405788519Num uniques 1030; 2023-04-25 06:56:28,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-25 06:56:28,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-25 06:56:28,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682405788531, max timestamp=1682405788542Num uniques 1039; 2023-04-25 06:56:28,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-25 06:56:28,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682405788531, max timestamp=1682405788542Num uniques 1039; 2023-04-25 06:56:28,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682405788531, max timestamp=1682405788542Num uniques 1039; 2023-04-25 06:56:28,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-25 06:56:28,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-25 06:56:28,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682405788557, max timestamp=1682405788565Num uniques 1098; 2023-04-25 06:56:28,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-25 06:56:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682405788557, max timestamp=1682405788565Num uniques 1098; 2023-04-25 06:56:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-04-25 06:56:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682405788557, max timestamp=1682405788568Num uniques 1098; 2023-04-25 06:56:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-04-25 06:56:28,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682405788584, max timestamp=1682405788593Num uniques 1057; 2023-04-25 06:56:28,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-25 06:56:28,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682405788584, max timestamp=1682405788593Num uniques 1107; 2023-04-25 06:56:28,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-25 06:56:28,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682405788584, max timestamp=1682405788593Num uniques 1057; 2023-04-25 06:56:28,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-25 06:56:28,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682405788594, max timestamp=1682405788615Num uniques 1066; 2023-04-25 06:56:28,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-25 06:56:28,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1682405788594, max timestamp=1682405788615Num uniques 1066; 2023-04-25 06:56:28,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-25 06:56:28,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1682405788594, max timestamp=1682405788615Num uniques 1066; 2023-04-25 06:56:28,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-25 06:56:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1682405788625, max timestamp=1682405788638Num uniques 1075; 2023-04-25 06:56:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-04-25 06:56:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1682405788625, max timestamp=1682405788638Num uniques 1025; 2023-04-25 06:56:28,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=8010, suffix heap size=134568,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=175 2023-04-25 06:56:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1682405788625, max timestamp=1682405788638Num uniques 1025; 2023-04-25 06:56:28,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=8010, suffix heap size=134568,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=175 2023-04-25 06:56:28,649 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/4bcfe6faa9d24683a756c11144d28592 2023-04-25 06:56:28,657 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/6adeef89d437452786fb49f6f4ab1947 2023-04-25 06:56:28,661 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/afab958b0887452ab885dafd2f3e4ef7 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/afab958b0887452ab885dafd2f3e4ef7 2023-04-25 06:56:28,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1682405788655, max timestamp=1682405788662Num uniques 1034; 2023-04-25 06:56:28,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=13004, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-04-25 06:56:28,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1682405788655, max timestamp=1682405788662Num uniques 1034; 2023-04-25 06:56:28,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=13004, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-04-25 06:56:28,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1682405788655, max timestamp=1682405788662Num uniques 1034; 2023-04-25 06:56:28,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=13004, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-04-25 06:56:28,667 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/afab958b0887452ab885dafd2f3e4ef7, entries=200, sequenceid=407, filesize=32.3 K 2023-04-25 06:56:28,668 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/4bcfe6faa9d24683a756c11144d28592 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4bcfe6faa9d24683a756c11144d28592 2023-04-25 06:56:28,673 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4bcfe6faa9d24683a756c11144d28592, entries=100, sequenceid=407, filesize=9.5 K 2023-04-25 06:56:28,674 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/6adeef89d437452786fb49f6f4ab1947 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6adeef89d437452786fb49f6f4ab1947 2023-04-25 06:56:28,677 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6adeef89d437452786fb49f6f4ab1947, entries=100, sequenceid=407, filesize=9.5 K 2023-04-25 06:56:28,678 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=100.63 KB/103050 for e6358a6a069ce34b04034f9d45664d77 in 888ms, sequenceid=407, compaction requested=true 2023-04-25 06:56:28,678 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:28,678 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:28,679 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,679 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,680 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:28,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=107.34 KB heapSize=293.58 KB 2023-04-25 06:56:28,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:28,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:28,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:28,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:28,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:28,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 06:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6005737f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:28,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:28,683 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,685 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:28,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,686 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:28,687 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:28,687 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:28,690 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:28,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:28,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:28,690 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:28,690 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:28,692 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,692 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:28,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,692 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:28,692 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,693 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257dc2903d6e8e4fe59ec4c5e801748298 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257dc2903d6e8e4fe59ec4c5e801748298 2023-04-25 06:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,695 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405848693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405848695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405848696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405848697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,700 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:28,700 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:28,701 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:28,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:28,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:28,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:28,701 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:28,701 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:28,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:28,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,701 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:28,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405848799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,801 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:28,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:28,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:28,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405848801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405848801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:28,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:28,805 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:28,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:28,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:28,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:28,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:28,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405848802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,805 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,808 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:28,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,808 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:28,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:28,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:28,811 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fa009f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:28,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:28,813 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,814 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:28,814 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,815 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:28,816 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:28,820 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:28,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:28,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:28,820 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:28,820 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:28,823 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,823 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:28,823 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,825 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,826 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,827 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:28,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,828 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:28,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,828 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,828 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:28,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:28,828 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:28,928 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:28,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:28,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:28,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:28,932 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:28,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:28,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:28,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:28,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,932 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,934 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:28,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:28,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,934 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:28,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:28,938 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6ee028[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:28,940 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:28,940 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,942 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:28,942 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,942 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:28,943 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:28,947 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:28,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:28,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:28,947 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:28,947 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:28,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,948 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:28,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,949 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:28,950 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:28,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,952 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:28,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:28,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:28,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:28,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:28,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:28,953 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:28,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:28,953 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:28,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:28,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:28,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:28,953 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:28,953 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:29,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405849002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405849005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405849006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405849007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,054 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:29,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:29,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:29,057 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:29,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:29,057 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:29,058 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,058 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,062 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,062 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:29,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:29,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:29,066 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7650476[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:29,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:29,068 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,069 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:29,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,069 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:29,071 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:29,074 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:29,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:29,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:29,074 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:29,074 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:29,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,076 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,077 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,077 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,078 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,080 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:29,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:29,081 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:29,081 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,081 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,082 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,082 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:29,099 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=867, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/37ade06994324e8dbee597f71ab8fb67 2023-04-25 06:56:29,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=867 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/a9c9dc5c99e047e5af99774ff3987478 2023-04-25 06:56:29,181 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:29,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:29,184 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:29,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:29,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:29,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,185 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,186 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,186 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:29,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:29,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:29,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:29,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:29,188 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:29,190 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:29,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cb0116e[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:29,192 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:29,192 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,194 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,194 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:29,194 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:29,194 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:29,194 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,195 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:29,196 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:29,199 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:29,199 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:29,199 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:29,199 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:29,199 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:29,201 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,201 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,201 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,202 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,203 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,205 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,205 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,205 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:29,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:29,205 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,206 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,206 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,206 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,206 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,206 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,206 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:29,205 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,206 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:29,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:29,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,306 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:29,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:29,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:29,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405849307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:29,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:29,310 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:29,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:29,310 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:29,311 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,311 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,313 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,313 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,313 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,313 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405849311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405849312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405849312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:29,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:29,315 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,315 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:29,315 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:29,315 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:29,317 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cbe1e9f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:29,318 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:29,318 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,319 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,320 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:29,320 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:29,321 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:29,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:29,323 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:29,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:29,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:29,323 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:29,323 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:29,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,331 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,331 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,333 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,335 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,335 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,335 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,335 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:29,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:29,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:29,335 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,335 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,336 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:29,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,336 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:29,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,336 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,436 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:29,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:29,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:29,441 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:29,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:29,441 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:29,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,442 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,443 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,443 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,444 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:29,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:29,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:29,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:29,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:29,445 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,446 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:29,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44a9c680[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:29,448 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:29,448 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,453 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:29,454 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:29,454 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:29,454 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,454 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:29,455 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,455 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,455 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,455 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:29,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:29,458 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:29,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:29,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:29,458 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:29,458 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:29,460 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,460 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,460 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:29,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:29,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,461 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:29,462 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,464 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,464 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:29,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:29,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:29,464 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:29,464 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:29,465 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:29,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,465 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:29,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:29,465 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=867 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/03ad69343e2e4cc69554db78c27bf484 2023-04-25 06:56:29,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/37ade06994324e8dbee597f71ab8fb67 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/37ade06994324e8dbee597f71ab8fb67 2023-04-25 06:56:29,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/37ade06994324e8dbee597f71ab8fb67, entries=200, sequenceid=867, filesize=32.3 K 2023-04-25 06:56:29,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/a9c9dc5c99e047e5af99774ff3987478 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a9c9dc5c99e047e5af99774ff3987478 2023-04-25 06:56:29,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a9c9dc5c99e047e5af99774ff3987478, entries=150, sequenceid=867, filesize=11.9 K 2023-04-25 06:56:29,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/03ad69343e2e4cc69554db78c27bf484 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/03ad69343e2e4cc69554db78c27bf484 2023-04-25 06:56:29,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/03ad69343e2e4cc69554db78c27bf484, entries=150, sequenceid=867, filesize=11.9 K 2023-04-25 06:56:29,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~293.53 KB/300576, currentSize=73.80 KB/75570 for e6358a6a069ce34b04034f9d45664d77 in 857ms, sequenceid=867, compaction requested=true 2023-04-25 06:56:29,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:29,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:29,537 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:56:29,537 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:56:29,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:29,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:29,538 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 171136 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:56:29,539 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/A is initiating minor compaction (all files) 2023-04-25 06:56:29,539 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/A in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,539 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/67cb8ffa91214b428843bf994d771828, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/98f774e507544397a989e5b86dc67ec1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b3e502ea9650467aa144e7749c581784, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b8147ccbfd1248d9b76737fdf5e63ae9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/afab958b0887452ab885dafd2f3e4ef7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/37ade06994324e8dbee597f71ab8fb67] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=167.1 K 2023-04-25 06:56:29,539 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71456 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:56:29,539 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/B is initiating minor compaction (all files) 2023-04-25 06:56:29,539 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/B in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,539 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/fecd79c94ed5471bb2892cf976608f20, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/5bcc87e1dff34947901f766aaa81043c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/1dd96b3329b94094b2932c943273ac8c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/23ab25bc46224497a9d7b061a763726a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4bcfe6faa9d24683a756c11144d28592, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a9c9dc5c99e047e5af99774ff3987478] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=69.8 K 2023-04-25 06:56:29,539 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 67cb8ffa91214b428843bf994d771828, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682405783544 2023-04-25 06:56:29,539 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting fecd79c94ed5471bb2892cf976608f20, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682405783544 2023-04-25 06:56:29,540 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 98f774e507544397a989e5b86dc67ec1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682405783868 2023-04-25 06:56:29,540 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5bcc87e1dff34947901f766aaa81043c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682405783868 2023-04-25 06:56:29,540 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b3e502ea9650467aa144e7749c581784, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682405784502 2023-04-25 06:56:29,541 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 1dd96b3329b94094b2932c943273ac8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682405784502 2023-04-25 06:56:29,541 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting b8147ccbfd1248d9b76737fdf5e63ae9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1682405785633 2023-04-25 06:56:29,541 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 23ab25bc46224497a9d7b061a763726a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1682405785633 2023-04-25 06:56:29,541 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting afab958b0887452ab885dafd2f3e4ef7, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682405787766 2023-04-25 06:56:29,541 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 4bcfe6faa9d24683a756c11144d28592, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682405787782 2023-04-25 06:56:29,541 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] compactions.Compactor(207): Compacting 37ade06994324e8dbee597f71ab8fb67, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1682405788657 2023-04-25 06:56:29,541 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a9c9dc5c99e047e5af99774ff3987478, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1682405788674 2023-04-25 06:56:29,551 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#B#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:29,565 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:29,566 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#A#compaction#557 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:29,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:29,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:29,579 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:29,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:29,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:29,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,580 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,583 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:29,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:29,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,583 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:29,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:29,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:29,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:29,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:29,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:29,588 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,588 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:29,588 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:56:29,588 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:29,588 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:29,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:29,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:29,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:29,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:29,620 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254de70baa4ffb423e91c7812c1847a85f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254de70baa4ffb423e91c7812c1847a85f 2023-04-25 06:56:29,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:29,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:29,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405849834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405849837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405849837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405849838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405849939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405849941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405849941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405849942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:29,978 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/49b8805451104fd983589fe65cf8bdea as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/49b8805451104fd983589fe65cf8bdea 2023-04-25 06:56:29,982 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/B of e6358a6a069ce34b04034f9d45664d77 into 49b8805451104fd983589fe65cf8bdea(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:29,982 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:29,982 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/B, priority=10, startTime=1682405789537; duration=0sec 2023-04-25 06:56:29,982 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 06:56:29,982 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 06:56:29,984 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71422 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 06:56:29,984 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1912): e6358a6a069ce34b04034f9d45664d77/C is initiating minor compaction (all files) 2023-04-25 06:56:29,984 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e6358a6a069ce34b04034f9d45664d77/C in TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:29,984 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a168d58afd8a4a57893ad47bc5bd67dd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/aed681f3758e440caaceba9a2c929c30, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5c3ad48a106749fe8f816d8ab85a479d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/893a5ba0e30f46198ccbf8c7a63d0fc3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6adeef89d437452786fb49f6f4ab1947, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/03ad69343e2e4cc69554db78c27bf484] into tmpdir=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp, totalSize=69.7 K 2023-04-25 06:56:29,985 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting a168d58afd8a4a57893ad47bc5bd67dd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682405783544 2023-04-25 06:56:29,985 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting aed681f3758e440caaceba9a2c929c30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682405783868 2023-04-25 06:56:29,985 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 5c3ad48a106749fe8f816d8ab85a479d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682405784502 2023-04-25 06:56:29,985 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 893a5ba0e30f46198ccbf8c7a63d0fc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1682405785633 2023-04-25 06:56:29,986 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 6adeef89d437452786fb49f6f4ab1947, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682405787782 2023-04-25 06:56:29,986 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] compactions.Compactor(207): Compacting 03ad69343e2e4cc69554db78c27bf484, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1682405788674 2023-04-25 06:56:29,995 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] throttle.PressureAwareThroughputController(145): e6358a6a069ce34b04034f9d45664d77#C#compaction#559 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 06:56:30,003 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/0a52d29a138c47159e59284470920aaa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/0a52d29a138c47159e59284470920aaa 2023-04-25 06:56:30,007 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/C of e6358a6a069ce34b04034f9d45664d77 into 0a52d29a138c47159e59284470920aaa(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:30,007 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:30,007 INFO [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/C, priority=10, startTime=1682405789537; duration=0sec 2023-04-25 06:56:30,007 DEBUG [RS:0;jenkins-hbase4:41021-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:30,016 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/09eb1f92addf449a8804ed2ae403adac as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/09eb1f92addf449a8804ed2ae403adac 2023-04-25 06:56:30,022 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e6358a6a069ce34b04034f9d45664d77/A of e6358a6a069ce34b04034f9d45664d77 into 09eb1f92addf449a8804ed2ae403adac(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 06:56:30,022 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:30,022 INFO [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., storeName=e6358a6a069ce34b04034f9d45664d77/A, priority=10, startTime=1682405789537; duration=0sec 2023-04-25 06:56:30,022 DEBUG [RS:0;jenkins-hbase4:41021-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 06:56:30,025 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=881, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/a6180e8c6f7c4fbfb9a12717bddcc272 2023-04-25 06:56:30,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39948 deadline: 1682405850067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,070 DEBUG [Thread-3618] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77., hostname=jenkins-hbase4.apache.org,41021,1682405622759, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 06:56:30,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405850143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405850145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405850145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405850147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,434 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=881 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e978d1bcd7754a03881d29dc88d7fcd5 2023-04-25 06:56:30,444 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=881 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/7e5b1069fec9462fa919160ea799555d 2023-04-25 06:56:30,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405850445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,448 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/a6180e8c6f7c4fbfb9a12717bddcc272 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a6180e8c6f7c4fbfb9a12717bddcc272 2023-04-25 06:56:30,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405850448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405850448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405850450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,452 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a6180e8c6f7c4fbfb9a12717bddcc272, entries=150, sequenceid=881, filesize=25.4 K 2023-04-25 06:56:30,453 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/e978d1bcd7754a03881d29dc88d7fcd5 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e978d1bcd7754a03881d29dc88d7fcd5 2023-04-25 06:56:30,457 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e978d1bcd7754a03881d29dc88d7fcd5, entries=150, sequenceid=881, filesize=11.9 K 2023-04-25 06:56:30,458 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/7e5b1069fec9462fa919160ea799555d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e5b1069fec9462fa919160ea799555d 2023-04-25 06:56:30,461 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e5b1069fec9462fa919160ea799555d, entries=150, sequenceid=881, filesize=11.9 K 2023-04-25 06:56:30,461 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e6358a6a069ce34b04034f9d45664d77 in 873ms, sequenceid=881, compaction requested=false 2023-04-25 06:56:30,461 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:30,461 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,463 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:30,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:30,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bc02cd7[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:30,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:30,465 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,466 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:30,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,467 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:30,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:30,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:30,468 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:30,468 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:30,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:30,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:30,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:30,471 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:30,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:30,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:30,471 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:30,471 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:30,474 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,474 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:30,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,474 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:30,475 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,476 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:30,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:30,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:30,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:30,479 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:30,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:30,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:30,480 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:30,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:30,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:30,480 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:30,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:30,480 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:30,480 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:30,581 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:30,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:30,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:30,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:30,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:30,584 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:30,584 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:30,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:30,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:30,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,585 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:30,586 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:30,586 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:30,587 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,587 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:30,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:30,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:30,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:30,588 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:30,588 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:30,589 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 06:56:30,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:30,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:30,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:30,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:30,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:30,589 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:30,611 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257b6534ec60244bf8ace19b0b0314bd5f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257b6534ec60244bf8ace19b0b0314bd5f 2023-04-25 06:56:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:30,952 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:30,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405850963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405850963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405850963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405850964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,017 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=902, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/3854f6d9ba104ac780d6a4a0e04fe29b 2023-04-25 06:56:31,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405851067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405851067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405851067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405851067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405851270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405851270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405851272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405851272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,427 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/b7e817ba29d7458eb5b0726c232bf639 2023-04-25 06:56:31,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405851573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405851573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405851574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:31,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405851576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,836 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/54b36439479e41c5a09161a2e96d0713 2023-04-25 06:56:31,840 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/3854f6d9ba104ac780d6a4a0e04fe29b as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/3854f6d9ba104ac780d6a4a0e04fe29b 2023-04-25 06:56:31,843 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/3854f6d9ba104ac780d6a4a0e04fe29b, entries=150, sequenceid=902, filesize=25.4 K 2023-04-25 06:56:31,844 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/b7e817ba29d7458eb5b0726c232bf639 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b7e817ba29d7458eb5b0726c232bf639 2023-04-25 06:56:31,847 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b7e817ba29d7458eb5b0726c232bf639, entries=150, sequenceid=902, filesize=11.9 K 2023-04-25 06:56:31,848 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/54b36439479e41c5a09161a2e96d0713 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/54b36439479e41c5a09161a2e96d0713 2023-04-25 06:56:31,851 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/54b36439479e41c5a09161a2e96d0713, entries=150, sequenceid=902, filesize=11.9 K 2023-04-25 06:56:31,852 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e6358a6a069ce34b04034f9d45664d77 in 1264ms, sequenceid=902, compaction requested=true 2023-04-25 06:56:31,852 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:31,852 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:31,853 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:31,854 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:31,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,855 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:31,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:31,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27376ba0[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:31,856 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:31,856 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,859 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:31,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,859 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:31,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:31,859 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:31,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:31,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:31,860 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,860 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:31,863 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:31,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:31,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:31,863 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:31,863 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:31,865 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,865 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,865 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:31,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:31,865 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,867 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,869 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:31,870 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:31,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:31,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:31,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:31,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:31,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:31,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:31,870 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:31,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:31,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:31,870 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,870 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:31,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:31,970 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:31,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:31,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:31,974 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:31,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:31,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:31,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:31,974 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,975 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:31,976 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:31,976 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:31,977 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:31,977 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:31,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:31,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:31,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:31,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:31,978 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:31,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:31,979 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 06:56:31,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:31,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:31,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:31,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:31,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:31,979 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:31,996 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042599656cfbba4d4226a02eea96b3365e3a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042599656cfbba4d4226a02eea96b3365e3a 2023-04-25 06:56:32,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:32,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:32,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405852091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405852096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405852096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405852097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405852197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405852202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405852202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405852202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405852400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,406 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=916, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/16bbeebed31d4c87903f1c7f6307a4dd 2023-04-25 06:56:32,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405852405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405852405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405852406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405852703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405852708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405852710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:32,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405852710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:32,817 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/b3b3d4a0ba224cd992f34a6da5937bfa 2023-04-25 06:56:33,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:33,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39940 deadline: 1682405853210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:33,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39894 deadline: 1682405853213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41021] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39904 deadline: 1682405853213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 06:56:33,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41021] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39930 deadline: 1682405853216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,227 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/bf1af43eeb0f463380efb0705ab964e0 2023-04-25 06:56:33,231 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/16bbeebed31d4c87903f1c7f6307a4dd as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/16bbeebed31d4c87903f1c7f6307a4dd 2023-04-25 06:56:33,235 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/16bbeebed31d4c87903f1c7f6307a4dd, entries=150, sequenceid=916, filesize=25.4 K 2023-04-25 06:56:33,236 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/b3b3d4a0ba224cd992f34a6da5937bfa as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b3b3d4a0ba224cd992f34a6da5937bfa 2023-04-25 06:56:33,248 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b3b3d4a0ba224cd992f34a6da5937bfa, entries=150, sequenceid=916, filesize=11.9 K 2023-04-25 06:56:33,249 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/bf1af43eeb0f463380efb0705ab964e0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/bf1af43eeb0f463380efb0705ab964e0 2023-04-25 06:56:33,253 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/bf1af43eeb0f463380efb0705ab964e0, entries=150, sequenceid=916, filesize=11.9 K 2023-04-25 06:56:33,253 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for e6358a6a069ce34b04034f9d45664d77 in 1274ms, sequenceid=916, compaction requested=true 2023-04-25 06:56:33,254 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:33,254 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:33,255 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,255 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:33,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:33,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b31c33f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:33,257 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:33,257 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,258 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:33,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:33,258 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:33,258 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,258 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:33,262 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:33,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:33,262 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:33,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:33,264 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:33,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:33,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:33,264 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:33,264 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:33,266 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,266 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:33,266 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:33,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:33,267 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:33,268 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:33,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:33,277 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:33,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:33,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:33,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:33,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:33,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:33,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:33,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:33,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,277 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:33,278 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:33,277 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:33,278 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:33,378 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:33,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:33,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 06:56:33,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 06:56:33,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 06:56:33,382 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 06:56:33,382 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:33,382 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 06:56:33,382 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 06:56:33,382 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,382 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:33,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,385 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:33,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:33,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:33,385 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 06:56:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:33,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 06:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:33,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 06:56:33,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 06:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 06:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 06:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 06:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 06:56:33,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 06:56:33,387 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. started... 2023-04-25 06:56:33,387 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:33,387 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:33,404 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304258b4d5ee18115454da983bbeca1619776 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258b4d5ee18115454da983bbeca1619776 2023-04-25 06:56:33,771 DEBUG [Thread-3633] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35c3cca2 to 127.0.0.1:60416 2023-04-25 06:56:33,771 DEBUG [Thread-3633] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:33,772 DEBUG [Thread-3635] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b1973b9 to 127.0.0.1:60416 2023-04-25 06:56:33,772 DEBUG [Thread-3635] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:33,772 DEBUG [Thread-3627] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f1adb60 to 127.0.0.1:60416 2023-04-25 06:56:33,772 DEBUG [Thread-3627] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:33,774 DEBUG [Thread-3631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43ea1da2 to 127.0.0.1:60416 2023-04-25 06:56:33,774 DEBUG [Thread-3631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:33,775 DEBUG [Thread-3629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ed73d5e to 127.0.0.1:60416 2023-04-25 06:56:33,775 DEBUG [Thread-3629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:33,810 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=935, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/189e642a5b974de1bda9207f9a7db6f0 2023-04-25 06:56:34,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41021] regionserver.HRegion(9158): Flush requested on e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:34,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. as already flushing 2023-04-25 06:56:34,216 DEBUG [Thread-3624] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a4acea2 to 127.0.0.1:60416 2023-04-25 06:56:34,216 DEBUG [Thread-3620] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55d0f313 to 127.0.0.1:60416 2023-04-25 06:56:34,216 DEBUG [Thread-3624] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:34,216 DEBUG [Thread-3620] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:34,218 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=935 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/cbe9b6de36104c64a97d54132e5ef1b2 2023-04-25 06:56:34,218 DEBUG [Thread-3622] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09d61148 to 127.0.0.1:60416 2023-04-25 06:56:34,218 DEBUG [Thread-3622] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:34,219 DEBUG [Thread-3616] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68d563c7 to 127.0.0.1:60416 2023-04-25 06:56:34,220 DEBUG [Thread-3616] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:34,626 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=935 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/da8d5804e26346b38c51ba01535f933f 2023-04-25 06:56:34,629 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/189e642a5b974de1bda9207f9a7db6f0 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/189e642a5b974de1bda9207f9a7db6f0 2023-04-25 06:56:34,632 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/189e642a5b974de1bda9207f9a7db6f0, entries=150, sequenceid=935, filesize=25.4 K 2023-04-25 06:56:34,632 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/cbe9b6de36104c64a97d54132e5ef1b2 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/cbe9b6de36104c64a97d54132e5ef1b2 2023-04-25 06:56:34,635 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/cbe9b6de36104c64a97d54132e5ef1b2, entries=150, sequenceid=935, filesize=11.9 K 2023-04-25 06:56:34,636 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/da8d5804e26346b38c51ba01535f933f as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/da8d5804e26346b38c51ba01535f933f 2023-04-25 06:56:34,638 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/da8d5804e26346b38c51ba01535f933f, entries=150, sequenceid=935, filesize=11.9 K 2023-04-25 06:56:34,639 INFO [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=26.84 KB/27480 for e6358a6a069ce34b04034f9d45664d77 in 1252ms, sequenceid=935, compaction requested=true 2023-04-25 06:56:34,639 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:34,639 DEBUG [rs(jenkins-hbase4.apache.org,41021,1682405622759)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 06:56:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 06:56:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 06:56:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 06:56:34,640 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,640 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 06:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:34,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41021,1682405622759' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 06:56:34,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6126951f[Count = 0] remaining members to acquire global barrier 2023-04-25 06:56:34,642 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 06:56:34,642 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,643 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 06:56:34,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 06:56:34,643 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41021,1682405622759' in zk 2023-04-25 06:56:34,643 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,643 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 06:56:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 06:56:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:34,644 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,41021,1682405622759' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 06:56:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41021,1682405622759': 2023-04-25 06:56:34,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41021,1682405622759' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 06:56:34,646 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 06:56:34,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 06:56:34,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 06:56:34,646 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 06:56:34,646 INFO [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 06:56:34,648 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:34,648 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 06:56:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 06:56:34,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:34,648 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 06:56:34,649 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 06:56:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:34,650 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 06:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 06:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 06:56:34,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 06:56:34,651 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 06:56:34,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 06:56:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 06:56:34,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 06:56:34,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 06:56:34,651 DEBUG [(jenkins-hbase4.apache.org,36703,1682405621453)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 06:56:34,652 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 06:56:34,652 DEBUG [Thread-3626] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 06:56:34,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 06:56:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 06:56:34,652 DEBUG [Thread-3626] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 06:56:34,752 DEBUG [Thread-3626] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 06:56:34,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 06:56:40,166 DEBUG [Thread-3618] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b13e8aa to 127.0.0.1:60416 2023-04-25 06:56:40,166 DEBUG [Thread-3618] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:40,166 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 06:56:40,166 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 186 2023-04-25 06:56:40,166 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 184 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 212 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(394): wrote 191 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 4469 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 4261 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 4175 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 4459 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(398): read 4289 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 06:56:40,167 INFO [Listener at localhost/41663] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 06:56:40,167 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x050d2537 to 127.0.0.1:60416 2023-04-25 06:56:40,167 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:40,168 INFO [Listener at localhost/41663] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 06:56:40,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 06:56:40,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 06:56:40,171 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405800171"}]},"ts":"1682405800171"} 2023-04-25 06:56:40,172 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 06:56:40,177 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 06:56:40,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, UNASSIGN}] 2023-04-25 06:56:40,179 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, UNASSIGN 2023-04-25 06:56:40,180 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:40,180 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405800180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682405800180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682405800180"}]},"ts":"1682405800180"} 2023-04-25 06:56:40,181 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759}] 2023-04-25 06:56:40,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 06:56:40,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e6358a6a069ce34b04034f9d45664d77, disabling compactions & flushes 2023-04-25 06:56:40,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. after waiting 0 ms 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:40,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e6358a6a069ce34b04034f9d45664d77 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=A 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=B 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e6358a6a069ce34b04034f9d45664d77, store=C 2023-04-25 06:56:40,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 06:56:40,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 06:56:40,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d69e3a0ef3ff4c688d0907cc43278768 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d69e3a0ef3ff4c688d0907cc43278768 2023-04-25 06:56:40,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 06:56:41,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=943, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/5238c5dc8c38497eaed4525710e2d4ed 2023-04-25 06:56:41,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 06:56:41,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=943 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/4c36937fae674902b6ebad43cd9d0fad 2023-04-25 06:56:41,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=943 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/95dafd5ed3e041dcbd17316dc4e50c11 2023-04-25 06:56:41,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/A/5238c5dc8c38497eaed4525710e2d4ed as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/5238c5dc8c38497eaed4525710e2d4ed 2023-04-25 06:56:41,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/5238c5dc8c38497eaed4525710e2d4ed, entries=150, sequenceid=943, filesize=25.4 K 2023-04-25 06:56:41,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/B/4c36937fae674902b6ebad43cd9d0fad as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4c36937fae674902b6ebad43cd9d0fad 2023-04-25 06:56:41,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4c36937fae674902b6ebad43cd9d0fad, entries=150, sequenceid=943, filesize=11.9 K 2023-04-25 06:56:41,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/.tmp/C/95dafd5ed3e041dcbd17316dc4e50c11 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/95dafd5ed3e041dcbd17316dc4e50c11 2023-04-25 06:56:41,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/95dafd5ed3e041dcbd17316dc4e50c11, entries=150, sequenceid=943, filesize=11.9 K 2023-04-25 06:56:41,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for e6358a6a069ce34b04034f9d45664d77 in 1645ms, sequenceid=943, compaction requested=true 2023-04-25 06:56:42,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9adaa40bfab542f299e93373d9a29de4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6fd4c8789c2d46658d23d32318cae002, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/73ac22131f7249ec9dde9723f474097a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/acc27e0dbf5d4e8eb0e198c97fd112a4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7496170aac38474fa48bf138e06f7200, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/19fce01a35fe43d091a2bcd30645b07f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/93550532eb9148cea4ae6058bca98462, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/79831d8db1394d92b6c4ffa36265e4ef, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a0a095c88d7e406c9b77aa61792a0fed, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b781ebece9cb46e7a8db3369574ca0d3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/ad084ccca90844a793f4d32c26aa4797, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fa67943f761a422eb713548e069b2060, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/08bd17558bd440acac2d7a5a1e6a608d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fbaf317858ee431fb54e869c3335d79e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/845ba9f5cff8432bab6b3ced18cf19f0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6d65fa411e2d45b08a3f39c88ac0edd2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/12499782513442e9a1ee1841a81cfb78, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a472be23ee8449eaa8381b246e657a84, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/045a6e1c92094a87a448f01227da93ca, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7b6261e5579c495ebab226081f962b9b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9ee112f7e4a44cc399d29044f2945366, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/f976d6e59ab84365b7ed325e85d32e17, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/e8fc26e388c64ff8838c3c6abe638d14, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/67cb8ffa91214b428843bf994d771828, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/98f774e507544397a989e5b86dc67ec1, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b3e502ea9650467aa144e7749c581784, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b8147ccbfd1248d9b76737fdf5e63ae9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/afab958b0887452ab885dafd2f3e4ef7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/37ade06994324e8dbee597f71ab8fb67] to archive 2023-04-25 06:56:42,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:56:42,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9adaa40bfab542f299e93373d9a29de4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9adaa40bfab542f299e93373d9a29de4 2023-04-25 06:56:42,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6fd4c8789c2d46658d23d32318cae002 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6fd4c8789c2d46658d23d32318cae002 2023-04-25 06:56:42,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/73ac22131f7249ec9dde9723f474097a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/73ac22131f7249ec9dde9723f474097a 2023-04-25 06:56:42,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/acc27e0dbf5d4e8eb0e198c97fd112a4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/acc27e0dbf5d4e8eb0e198c97fd112a4 2023-04-25 06:56:42,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7496170aac38474fa48bf138e06f7200 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7496170aac38474fa48bf138e06f7200 2023-04-25 06:56:42,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/19fce01a35fe43d091a2bcd30645b07f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/19fce01a35fe43d091a2bcd30645b07f 2023-04-25 06:56:42,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/93550532eb9148cea4ae6058bca98462 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/93550532eb9148cea4ae6058bca98462 2023-04-25 06:56:42,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/79831d8db1394d92b6c4ffa36265e4ef to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/79831d8db1394d92b6c4ffa36265e4ef 2023-04-25 06:56:42,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a0a095c88d7e406c9b77aa61792a0fed to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a0a095c88d7e406c9b77aa61792a0fed 2023-04-25 06:56:42,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b781ebece9cb46e7a8db3369574ca0d3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b781ebece9cb46e7a8db3369574ca0d3 2023-04-25 06:56:42,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/ad084ccca90844a793f4d32c26aa4797 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/ad084ccca90844a793f4d32c26aa4797 2023-04-25 06:56:42,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fa67943f761a422eb713548e069b2060 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fa67943f761a422eb713548e069b2060 2023-04-25 06:56:42,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/08bd17558bd440acac2d7a5a1e6a608d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/08bd17558bd440acac2d7a5a1e6a608d 2023-04-25 06:56:42,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fbaf317858ee431fb54e869c3335d79e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/fbaf317858ee431fb54e869c3335d79e 2023-04-25 06:56:42,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/845ba9f5cff8432bab6b3ced18cf19f0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/845ba9f5cff8432bab6b3ced18cf19f0 2023-04-25 06:56:42,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6d65fa411e2d45b08a3f39c88ac0edd2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/6d65fa411e2d45b08a3f39c88ac0edd2 2023-04-25 06:56:42,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/12499782513442e9a1ee1841a81cfb78 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/12499782513442e9a1ee1841a81cfb78 2023-04-25 06:56:42,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a472be23ee8449eaa8381b246e657a84 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a472be23ee8449eaa8381b246e657a84 2023-04-25 06:56:42,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/045a6e1c92094a87a448f01227da93ca to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/045a6e1c92094a87a448f01227da93ca 2023-04-25 06:56:42,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7b6261e5579c495ebab226081f962b9b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/7b6261e5579c495ebab226081f962b9b 2023-04-25 06:56:42,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9ee112f7e4a44cc399d29044f2945366 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/9ee112f7e4a44cc399d29044f2945366 2023-04-25 06:56:42,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/f976d6e59ab84365b7ed325e85d32e17 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/f976d6e59ab84365b7ed325e85d32e17 2023-04-25 06:56:42,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/e8fc26e388c64ff8838c3c6abe638d14 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/e8fc26e388c64ff8838c3c6abe638d14 2023-04-25 06:56:42,023 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/67cb8ffa91214b428843bf994d771828 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/67cb8ffa91214b428843bf994d771828 2023-04-25 06:56:42,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/98f774e507544397a989e5b86dc67ec1 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/98f774e507544397a989e5b86dc67ec1 2023-04-25 06:56:42,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b3e502ea9650467aa144e7749c581784 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b3e502ea9650467aa144e7749c581784 2023-04-25 06:56:42,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b8147ccbfd1248d9b76737fdf5e63ae9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/b8147ccbfd1248d9b76737fdf5e63ae9 2023-04-25 06:56:42,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/afab958b0887452ab885dafd2f3e4ef7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/afab958b0887452ab885dafd2f3e4ef7 2023-04-25 06:56:42,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/37ade06994324e8dbee597f71ab8fb67 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/37ade06994324e8dbee597f71ab8fb67 2023-04-25 06:56:42,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9dbbfb0ecac74d62bc798bc551870051, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a41c05d64cf8476b83056f0a16998e49, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/2efa23b4d6064025b5e375f5d44fdccf, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6bfe350ddf1c434fadb24721ecffd856, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a8b023463a224282bab1c0711a76e737, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9436a73d3ca542da9940c639076d5f6e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6c9a6a14f5194cf4994b6022f9a47b28, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/8607862ed43b4a819e4b2dda3ea4b57e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e7adfc08dac349c49987d43d9309e0ee, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/ed9fe92af43c4b348e0466bf9ca0cb6c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/f4d1e308bb7f4a0a9a6d33aaf35a566f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/0112556ad06e4f04a24d644d48f9dcbe, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/7dbfb36bae1947f3a8c3122efcb7a5e9, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c3eb1538705d4a38a980a303894862e4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6633fecf6ddd45b0a77e37fd5b69100c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/771ef43f41164926a21624ce5e00ad99, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c53ecd35a1924ca7ad09a245840a9b32, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/292b9eb770944813ae6c1d23aeb37442, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9756781601641d3a9dea13444e982f2, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9b92c498561406ea9e0db59cd2693ff, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/de3607b53639429da2633ffdc6e412d4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e3fe11a539344c849f2bffe9e995c2aa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/fecd79c94ed5471bb2892cf976608f20, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b9b9cb6b1a5b44228ddd7d34c94e8caa, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/5bcc87e1dff34947901f766aaa81043c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/1dd96b3329b94094b2932c943273ac8c, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/23ab25bc46224497a9d7b061a763726a, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4bcfe6faa9d24683a756c11144d28592, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a9c9dc5c99e047e5af99774ff3987478] to archive 2023-04-25 06:56:42,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:56:42,032 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9dbbfb0ecac74d62bc798bc551870051 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9dbbfb0ecac74d62bc798bc551870051 2023-04-25 06:56:42,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a41c05d64cf8476b83056f0a16998e49 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a41c05d64cf8476b83056f0a16998e49 2023-04-25 06:56:42,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/2efa23b4d6064025b5e375f5d44fdccf to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/2efa23b4d6064025b5e375f5d44fdccf 2023-04-25 06:56:42,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6bfe350ddf1c434fadb24721ecffd856 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6bfe350ddf1c434fadb24721ecffd856 2023-04-25 06:56:42,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a8b023463a224282bab1c0711a76e737 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a8b023463a224282bab1c0711a76e737 2023-04-25 06:56:42,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9436a73d3ca542da9940c639076d5f6e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/9436a73d3ca542da9940c639076d5f6e 2023-04-25 06:56:42,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6c9a6a14f5194cf4994b6022f9a47b28 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6c9a6a14f5194cf4994b6022f9a47b28 2023-04-25 06:56:42,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/8607862ed43b4a819e4b2dda3ea4b57e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/8607862ed43b4a819e4b2dda3ea4b57e 2023-04-25 06:56:42,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e7adfc08dac349c49987d43d9309e0ee to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e7adfc08dac349c49987d43d9309e0ee 2023-04-25 06:56:42,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/ed9fe92af43c4b348e0466bf9ca0cb6c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/ed9fe92af43c4b348e0466bf9ca0cb6c 2023-04-25 06:56:42,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/f4d1e308bb7f4a0a9a6d33aaf35a566f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/f4d1e308bb7f4a0a9a6d33aaf35a566f 2023-04-25 06:56:42,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/0112556ad06e4f04a24d644d48f9dcbe to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/0112556ad06e4f04a24d644d48f9dcbe 2023-04-25 06:56:42,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/7dbfb36bae1947f3a8c3122efcb7a5e9 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/7dbfb36bae1947f3a8c3122efcb7a5e9 2023-04-25 06:56:42,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c3eb1538705d4a38a980a303894862e4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c3eb1538705d4a38a980a303894862e4 2023-04-25 06:56:42,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6633fecf6ddd45b0a77e37fd5b69100c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/6633fecf6ddd45b0a77e37fd5b69100c 2023-04-25 06:56:42,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/771ef43f41164926a21624ce5e00ad99 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/771ef43f41164926a21624ce5e00ad99 2023-04-25 06:56:42,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c53ecd35a1924ca7ad09a245840a9b32 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/c53ecd35a1924ca7ad09a245840a9b32 2023-04-25 06:56:42,046 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/292b9eb770944813ae6c1d23aeb37442 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/292b9eb770944813ae6c1d23aeb37442 2023-04-25 06:56:42,047 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9756781601641d3a9dea13444e982f2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9756781601641d3a9dea13444e982f2 2023-04-25 06:56:42,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9b92c498561406ea9e0db59cd2693ff to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e9b92c498561406ea9e0db59cd2693ff 2023-04-25 06:56:42,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/de3607b53639429da2633ffdc6e412d4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/de3607b53639429da2633ffdc6e412d4 2023-04-25 06:56:42,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e3fe11a539344c849f2bffe9e995c2aa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e3fe11a539344c849f2bffe9e995c2aa 2023-04-25 06:56:42,050 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/fecd79c94ed5471bb2892cf976608f20 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/fecd79c94ed5471bb2892cf976608f20 2023-04-25 06:56:42,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b9b9cb6b1a5b44228ddd7d34c94e8caa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b9b9cb6b1a5b44228ddd7d34c94e8caa 2023-04-25 06:56:42,052 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/5bcc87e1dff34947901f766aaa81043c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/5bcc87e1dff34947901f766aaa81043c 2023-04-25 06:56:42,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/1dd96b3329b94094b2932c943273ac8c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/1dd96b3329b94094b2932c943273ac8c 2023-04-25 06:56:42,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/23ab25bc46224497a9d7b061a763726a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/23ab25bc46224497a9d7b061a763726a 2023-04-25 06:56:42,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4bcfe6faa9d24683a756c11144d28592 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4bcfe6faa9d24683a756c11144d28592 2023-04-25 06:56:42,056 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a9c9dc5c99e047e5af99774ff3987478 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/a9c9dc5c99e047e5af99774ff3987478 2023-04-25 06:56:42,059 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d4b0be4a0d3b4841991fcea4c0079eac, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dbdb692e10754cec91156ba9a9577524, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/e7677a17fee24b91b549ce31c6e91c5f, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1b010923088748868aa3fe2c5146c59d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d60a93816372438baf9f872a15b1fc85, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/ea5230e047144784b4f9e4bae37a2a6b, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1effbfe44fe64c9d8fd4076a58315804, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/4b4ccb4429884d899dec3912c336bd7e, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dc9140e428e548f7a0c2b9e18f9e81d7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/831c883207464bdaaf012a1cc0de32d0, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3e2575af41274d90a381d23d11902c1d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5a7c8fb1d37c4c09a8e43a33a5b5f001, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a09d8a4a121841d8b516f286df7699dc, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5b51d93291b14009ab5b0d389b2d3589, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3018094a21bd44af94e1c7c8e960efa7, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1c3c59de0d7c4c48848626665191df31, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6332d342fe0f4418b2be944c178ab4e6, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/01e44e0da5ad4a4496cfff0abc550335, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/727ef22ad0e0443ab2a184d2b268a3f4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e28931c5e5248ecb709f6690bc5a8d4, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/9aabf9c3187b4a8abb16ce8eae8b4b49, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a168d58afd8a4a57893ad47bc5bd67dd, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/b0cfa60029514a5b896212fc7fb2aeb5, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/aed681f3758e440caaceba9a2c929c30, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5c3ad48a106749fe8f816d8ab85a479d, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/893a5ba0e30f46198ccbf8c7a63d0fc3, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6adeef89d437452786fb49f6f4ab1947, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/03ad69343e2e4cc69554db78c27bf484] to archive 2023-04-25 06:56:42,060 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 06:56:42,061 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d4b0be4a0d3b4841991fcea4c0079eac to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d4b0be4a0d3b4841991fcea4c0079eac 2023-04-25 06:56:42,062 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dbdb692e10754cec91156ba9a9577524 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dbdb692e10754cec91156ba9a9577524 2023-04-25 06:56:42,063 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/e7677a17fee24b91b549ce31c6e91c5f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/e7677a17fee24b91b549ce31c6e91c5f 2023-04-25 06:56:42,064 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1b010923088748868aa3fe2c5146c59d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1b010923088748868aa3fe2c5146c59d 2023-04-25 06:56:42,065 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d60a93816372438baf9f872a15b1fc85 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/d60a93816372438baf9f872a15b1fc85 2023-04-25 06:56:42,065 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/ea5230e047144784b4f9e4bae37a2a6b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/ea5230e047144784b4f9e4bae37a2a6b 2023-04-25 06:56:42,066 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1effbfe44fe64c9d8fd4076a58315804 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1effbfe44fe64c9d8fd4076a58315804 2023-04-25 06:56:42,067 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/4b4ccb4429884d899dec3912c336bd7e to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/4b4ccb4429884d899dec3912c336bd7e 2023-04-25 06:56:42,068 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dc9140e428e548f7a0c2b9e18f9e81d7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/dc9140e428e548f7a0c2b9e18f9e81d7 2023-04-25 06:56:42,069 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/831c883207464bdaaf012a1cc0de32d0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/831c883207464bdaaf012a1cc0de32d0 2023-04-25 06:56:42,070 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3e2575af41274d90a381d23d11902c1d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3e2575af41274d90a381d23d11902c1d 2023-04-25 06:56:42,071 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5a7c8fb1d37c4c09a8e43a33a5b5f001 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5a7c8fb1d37c4c09a8e43a33a5b5f001 2023-04-25 06:56:42,072 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a09d8a4a121841d8b516f286df7699dc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a09d8a4a121841d8b516f286df7699dc 2023-04-25 06:56:42,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5b51d93291b14009ab5b0d389b2d3589 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5b51d93291b14009ab5b0d389b2d3589 2023-04-25 06:56:42,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3018094a21bd44af94e1c7c8e960efa7 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/3018094a21bd44af94e1c7c8e960efa7 2023-04-25 06:56:42,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1c3c59de0d7c4c48848626665191df31 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/1c3c59de0d7c4c48848626665191df31 2023-04-25 06:56:42,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6332d342fe0f4418b2be944c178ab4e6 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6332d342fe0f4418b2be944c178ab4e6 2023-04-25 06:56:42,076 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/01e44e0da5ad4a4496cfff0abc550335 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/01e44e0da5ad4a4496cfff0abc550335 2023-04-25 06:56:42,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/727ef22ad0e0443ab2a184d2b268a3f4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/727ef22ad0e0443ab2a184d2b268a3f4 2023-04-25 06:56:42,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e28931c5e5248ecb709f6690bc5a8d4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e28931c5e5248ecb709f6690bc5a8d4 2023-04-25 06:56:42,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/9aabf9c3187b4a8abb16ce8eae8b4b49 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/9aabf9c3187b4a8abb16ce8eae8b4b49 2023-04-25 06:56:42,080 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a168d58afd8a4a57893ad47bc5bd67dd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/a168d58afd8a4a57893ad47bc5bd67dd 2023-04-25 06:56:42,081 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/b0cfa60029514a5b896212fc7fb2aeb5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/b0cfa60029514a5b896212fc7fb2aeb5 2023-04-25 06:56:42,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/aed681f3758e440caaceba9a2c929c30 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/aed681f3758e440caaceba9a2c929c30 2023-04-25 06:56:42,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5c3ad48a106749fe8f816d8ab85a479d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/5c3ad48a106749fe8f816d8ab85a479d 2023-04-25 06:56:42,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/893a5ba0e30f46198ccbf8c7a63d0fc3 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/893a5ba0e30f46198ccbf8c7a63d0fc3 2023-04-25 06:56:42,084 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6adeef89d437452786fb49f6f4ab1947 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/6adeef89d437452786fb49f6f4ab1947 2023-04-25 06:56:42,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/03ad69343e2e4cc69554db78c27bf484 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/03ad69343e2e4cc69554db78c27bf484 2023-04-25 06:56:42,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/recovered.edits/946.seqid, newMaxSeqId=946, maxSeqId=4 2023-04-25 06:56:42,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77. 2023-04-25 06:56:42,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e6358a6a069ce34b04034f9d45664d77: 2023-04-25 06:56:42,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:42,091 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e6358a6a069ce34b04034f9d45664d77, regionState=CLOSED 2023-04-25 06:56:42,092 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682405802091"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682405802091"}]},"ts":"1682405802091"} 2023-04-25 06:56:42,093 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-25 06:56:42,093 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure e6358a6a069ce34b04034f9d45664d77, server=jenkins-hbase4.apache.org,41021,1682405622759 in 1.9110 sec 2023-04-25 06:56:42,094 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-25 06:56:42,094 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e6358a6a069ce34b04034f9d45664d77, UNASSIGN in 1.9150 sec 2023-04-25 06:56:42,095 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682405802095"}]},"ts":"1682405802095"} 2023-04-25 06:56:42,096 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 06:56:42,097 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 06:56:42,098 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9290 sec 2023-04-25 06:56:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 06:56:42,275 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-25 06:56:42,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 06:56:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:42,278 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:42,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-25 06:56:42,279 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:42,282 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:42,284 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C, FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/recovered.edits] 2023-04-25 06:56:42,286 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/09eb1f92addf449a8804ed2ae403adac to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/09eb1f92addf449a8804ed2ae403adac 2023-04-25 06:56:42,287 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/16bbeebed31d4c87903f1c7f6307a4dd to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/16bbeebed31d4c87903f1c7f6307a4dd 2023-04-25 06:56:42,288 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/189e642a5b974de1bda9207f9a7db6f0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/189e642a5b974de1bda9207f9a7db6f0 2023-04-25 06:56:42,289 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/3854f6d9ba104ac780d6a4a0e04fe29b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/3854f6d9ba104ac780d6a4a0e04fe29b 2023-04-25 06:56:42,290 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/5238c5dc8c38497eaed4525710e2d4ed to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/5238c5dc8c38497eaed4525710e2d4ed 2023-04-25 06:56:42,291 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a6180e8c6f7c4fbfb9a12717bddcc272 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/A/a6180e8c6f7c4fbfb9a12717bddcc272 2023-04-25 06:56:42,293 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/49b8805451104fd983589fe65cf8bdea to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/49b8805451104fd983589fe65cf8bdea 2023-04-25 06:56:42,294 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4c36937fae674902b6ebad43cd9d0fad to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/4c36937fae674902b6ebad43cd9d0fad 2023-04-25 06:56:42,295 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b3b3d4a0ba224cd992f34a6da5937bfa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b3b3d4a0ba224cd992f34a6da5937bfa 2023-04-25 06:56:42,297 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b7e817ba29d7458eb5b0726c232bf639 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/b7e817ba29d7458eb5b0726c232bf639 2023-04-25 06:56:42,298 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/cbe9b6de36104c64a97d54132e5ef1b2 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/cbe9b6de36104c64a97d54132e5ef1b2 2023-04-25 06:56:42,299 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e978d1bcd7754a03881d29dc88d7fcd5 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/B/e978d1bcd7754a03881d29dc88d7fcd5 2023-04-25 06:56:42,300 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/0a52d29a138c47159e59284470920aaa to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/0a52d29a138c47159e59284470920aaa 2023-04-25 06:56:42,301 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/54b36439479e41c5a09161a2e96d0713 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/54b36439479e41c5a09161a2e96d0713 2023-04-25 06:56:42,302 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e5b1069fec9462fa919160ea799555d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/7e5b1069fec9462fa919160ea799555d 2023-04-25 06:56:42,303 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/95dafd5ed3e041dcbd17316dc4e50c11 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/95dafd5ed3e041dcbd17316dc4e50c11 2023-04-25 06:56:42,304 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/bf1af43eeb0f463380efb0705ab964e0 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/bf1af43eeb0f463380efb0705ab964e0 2023-04-25 06:56:42,305 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/da8d5804e26346b38c51ba01535f933f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/C/da8d5804e26346b38c51ba01535f933f 2023-04-25 06:56:42,307 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/recovered.edits/946.seqid to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77/recovered.edits/946.seqid 2023-04-25 06:56:42,307 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/.tmp/data/default/TestAcidGuarantees/e6358a6a069ce34b04034f9d45664d77 2023-04-25 06:56:42,307 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 06:56:42,308 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 06:56:42,308 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 06:56:42,310 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042504925987441c495dbc7f151056a6b751 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042504925987441c495dbc7f151056a6b751 2023-04-25 06:56:42,311 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042507ea889148d54a16a08f0312bba56a5d to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042507ea889148d54a16a08f0312bba56a5d 2023-04-25 06:56:42,312 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042528d7b280e47e4b67a9c002b630d3dafc to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042528d7b280e47e4b67a9c002b630d3dafc 2023-04-25 06:56:42,313 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252b6c289ee2fc4965a91b98987a5e8546 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252b6c289ee2fc4965a91b98987a5e8546 2023-04-25 06:56:42,314 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425412ebe9caf15414da6cad8b9e2bd9a75 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425412ebe9caf15414da6cad8b9e2bd9a75 2023-04-25 06:56:42,315 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042541dcb9975bb64cdb9c183411f531be80 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042541dcb9975bb64cdb9c183411f531be80 2023-04-25 06:56:42,316 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254de70baa4ffb423e91c7812c1847a85f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254de70baa4ffb423e91c7812c1847a85f 2023-04-25 06:56:42,317 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042553cdbb225f3e4414a2cd0e916179139f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042553cdbb225f3e4414a2cd0e916179139f 2023-04-25 06:56:42,317 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425590b2333b8df4eabb6d66fa889231a95 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425590b2333b8df4eabb6d66fa889231a95 2023-04-25 06:56:42,318 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255dc7abe1be984e47aa4e02ac045a721b to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255dc7abe1be984e47aa4e02ac045a721b 2023-04-25 06:56:42,319 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042574c10ca700f0438c8f3756a925470d9c to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042574c10ca700f0438c8f3756a925470d9c 2023-04-25 06:56:42,320 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042575bda3f3538d49dda4ac5ad895b935ce to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042575bda3f3538d49dda4ac5ad895b935ce 2023-04-25 06:56:42,321 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257b6534ec60244bf8ace19b0b0314bd5f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257b6534ec60244bf8ace19b0b0314bd5f 2023-04-25 06:56:42,322 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257dc2903d6e8e4fe59ec4c5e801748298 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257dc2903d6e8e4fe59ec4c5e801748298 2023-04-25 06:56:42,323 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042580bbf71b753a4896b7e0d2c2c716471f to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042580bbf71b753a4896b7e0d2c2c716471f 2023-04-25 06:56:42,324 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258b4d5ee18115454da983bbeca1619776 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258b4d5ee18115454da983bbeca1619776 2023-04-25 06:56:42,325 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259171e579b277478d92f1992d278e58e4 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259171e579b277478d92f1992d278e58e4 2023-04-25 06:56:42,326 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042599656cfbba4d4226a02eea96b3365e3a to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042599656cfbba4d4226a02eea96b3365e3a 2023-04-25 06:56:42,326 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259e3d022d60164426bf1b1dbc3bb36fea to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259e3d022d60164426bf1b1dbc3bb36fea 2023-04-25 06:56:42,327 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b528e37dadaf43559a2bcf57397dd587 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b528e37dadaf43559a2bcf57397dd587 2023-04-25 06:56:42,328 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c768fdceaf204f8eaf448ca9471f01e8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c768fdceaf204f8eaf448ca9471f01e8 2023-04-25 06:56:42,329 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c954e9220d7b4e89b6866ad75c79f663 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c954e9220d7b4e89b6866ad75c79f663 2023-04-25 06:56:42,330 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cefad2309fb6456b8c15b7a53c0430f8 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cefad2309fb6456b8c15b7a53c0430f8 2023-04-25 06:56:42,331 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d1d41c7a1f36431dac7b77e0128c2322 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d1d41c7a1f36431dac7b77e0128c2322 2023-04-25 06:56:42,332 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d69e3a0ef3ff4c688d0907cc43278768 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d69e3a0ef3ff4c688d0907cc43278768 2023-04-25 06:56:42,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d9d1a910e56d4c40b1e0b78ca6e53a80 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d9d1a910e56d4c40b1e0b78ca6e53a80 2023-04-25 06:56:42,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ec3341237109439ea2393543ece8bd89 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ec3341237109439ea2393543ece8bd89 2023-04-25 06:56:42,334 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f4470a6368d041f4a12ffafe9370cf06 to hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f4470a6368d041f4a12ffafe9370cf06 2023-04-25 06:56:42,335 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 06:56:42,336 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:42,338 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 06:56:42,339 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 06:56:42,340 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:42,340 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 06:56:42,340 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682405802340"}]},"ts":"9223372036854775807"} 2023-04-25 06:56:42,341 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 06:56:42,341 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e6358a6a069ce34b04034f9d45664d77, NAME => 'TestAcidGuarantees,,1682405770680.e6358a6a069ce34b04034f9d45664d77.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 06:56:42,341 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 06:56:42,341 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682405802341"}]},"ts":"9223372036854775807"} 2023-04-25 06:56:42,342 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 06:56:42,344 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 06:56:42,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 68 msec 2023-04-25 06:56:42,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36703] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-25 06:56:42,380 INFO [Listener at localhost/41663] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-25 06:56:42,391 INFO [Listener at localhost/41663] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=524 (was 549), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=363 (was 389), ProcessCount=172 (was 172), AvailableMemoryMB=4646 (was 4530) - AvailableMemoryMB LEAK? - 2023-04-25 06:56:42,391 INFO [Listener at localhost/41663] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-25 06:56:42,392 INFO [Listener at localhost/41663] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 06:56:42,392 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d205425 to 127.0.0.1:60416 2023-04-25 06:56:42,392 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:42,392 DEBUG [Listener at localhost/41663] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-25 06:56:42,392 DEBUG [Listener at localhost/41663] util.JVMClusterUtil(257): Found active master hash=337198405, stopped=false 2023-04-25 06:56:42,393 INFO [Listener at localhost/41663] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:56:42,394 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-25 06:56:42,394 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-25 06:56:42,394 INFO [Listener at localhost/41663] procedure2.ProcedureExecutor(629): Stopping 2023-04-25 06:56:42,394 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:56:42,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 06:56:42,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 06:56:42,394 DEBUG [Listener at localhost/41663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2aeef597 to 127.0.0.1:60416 2023-04-25 06:56:42,395 DEBUG [Listener at localhost/41663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:42,395 INFO [Listener at localhost/41663] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41021,1682405622759' ***** 2023-04-25 06:56:42,395 INFO [Listener at localhost/41663] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-25 06:56:42,395 INFO [RS:0;jenkins-hbase4:41021] regionserver.HeapMemoryManager(220): Stopping 2023-04-25 06:56:42,395 INFO [RS:0;jenkins-hbase4:41021] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-25 06:56:42,395 INFO [RS:0;jenkins-hbase4:41021] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-25 06:56:42,395 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-25 06:56:42,396 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(3304): Received CLOSE for c31ed1c679bd183deae013f944f500ac 2023-04-25 06:56:42,396 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:42,396 DEBUG [RS:0;jenkins-hbase4:41021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ba5f95e to 127.0.0.1:60416 2023-04-25 06:56:42,397 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c31ed1c679bd183deae013f944f500ac, disabling compactions & flushes 2023-04-25 06:56:42,397 DEBUG [RS:0;jenkins-hbase4:41021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:42,397 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:56:42,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:56:42,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. after waiting 0 ms 2023-04-25 06:56:42,398 INFO [RS:0;jenkins-hbase4:41021] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-25 06:56:42,398 INFO [RS:0;jenkins-hbase4:41021] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-25 06:56:42,398 INFO [RS:0;jenkins-hbase4:41021] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-25 06:56:42,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:56:42,398 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-25 06:56:42,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c31ed1c679bd183deae013f944f500ac 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-25 06:56:42,398 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-25 06:56:42,398 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1479): Online Regions={c31ed1c679bd183deae013f944f500ac=hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac., 1588230740=hbase:meta,,1.1588230740} 2023-04-25 06:56:42,398 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-25 06:56:42,399 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-25 06:56:42,399 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-25 06:56:42,399 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-25 06:56:42,399 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-25 06:56:42,399 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-25 06:56:42,400 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1505): Waiting on 1588230740, c31ed1c679bd183deae013f944f500ac 2023-04-25 06:56:42,415 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/.tmp/info/b583907f8b3f4e528e74c823e78b813d 2023-04-25 06:56:42,419 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b583907f8b3f4e528e74c823e78b813d 2023-04-25 06:56:42,469 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-25 06:56:42,600 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1505): Waiting on 1588230740, c31ed1c679bd183deae013f944f500ac 2023-04-25 06:56:42,801 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1505): Waiting on 1588230740, c31ed1c679bd183deae013f944f500ac 2023-04-25 06:56:42,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/.tmp/info/616cc70a28d44b8db1ab9d05f8e32d72 2023-04-25 06:56:42,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/.tmp/info/616cc70a28d44b8db1ab9d05f8e32d72 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/info/616cc70a28d44b8db1ab9d05f8e32d72 2023-04-25 06:56:42,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/info/616cc70a28d44b8db1ab9d05f8e32d72, entries=2, sequenceid=6, filesize=4.8 K 2023-04-25 06:56:42,824 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/.tmp/rep_barrier/a2be02c081ca4dabbc2a29aaaf1ca922 2023-04-25 06:56:42,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c31ed1c679bd183deae013f944f500ac in 427ms, sequenceid=6, compaction requested=false 2023-04-25 06:56:42,829 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a2be02c081ca4dabbc2a29aaaf1ca922 2023-04-25 06:56:42,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/namespace/c31ed1c679bd183deae013f944f500ac/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-25 06:56:42,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:56:42,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c31ed1c679bd183deae013f944f500ac: 2023-04-25 06:56:42,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682405624808.c31ed1c679bd183deae013f944f500ac. 2023-04-25 06:56:43,001 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-25 06:56:43,186 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-25 06:56:43,186 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-25 06:56:43,201 DEBUG [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-25 06:56:43,240 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/.tmp/table/e4b229e679594c5c8abd90f8cc41f7c3 2023-04-25 06:56:43,243 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e4b229e679594c5c8abd90f8cc41f7c3 2023-04-25 06:56:43,243 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/.tmp/info/b583907f8b3f4e528e74c823e78b813d as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/info/b583907f8b3f4e528e74c823e78b813d 2023-04-25 06:56:43,246 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b583907f8b3f4e528e74c823e78b813d 2023-04-25 06:56:43,246 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/info/b583907f8b3f4e528e74c823e78b813d, entries=22, sequenceid=93, filesize=7.1 K 2023-04-25 06:56:43,248 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/.tmp/rep_barrier/a2be02c081ca4dabbc2a29aaaf1ca922 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/rep_barrier/a2be02c081ca4dabbc2a29aaaf1ca922 2023-04-25 06:56:43,250 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a2be02c081ca4dabbc2a29aaaf1ca922 2023-04-25 06:56:43,251 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/rep_barrier/a2be02c081ca4dabbc2a29aaaf1ca922, entries=6, sequenceid=93, filesize=5.5 K 2023-04-25 06:56:43,251 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/.tmp/table/e4b229e679594c5c8abd90f8cc41f7c3 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/table/e4b229e679594c5c8abd90f8cc41f7c3 2023-04-25 06:56:43,253 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e4b229e679594c5c8abd90f8cc41f7c3 2023-04-25 06:56:43,254 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/table/e4b229e679594c5c8abd90f8cc41f7c3, entries=9, sequenceid=93, filesize=5.5 K 2023-04-25 06:56:43,254 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 855ms, sequenceid=93, compaction requested=false 2023-04-25 06:56:43,262 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-25 06:56:43,262 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 06:56:43,263 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-25 06:56:43,263 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-25 06:56:43,263 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-25 06:56:43,401 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41021,1682405622759; all regions closed. 2023-04-25 06:56:43,410 DEBUG [RS:0;jenkins-hbase4:41021] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/oldWALs 2023-04-25 06:56:43,410 INFO [RS:0;jenkins-hbase4:41021] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41021%2C1682405622759.meta:.meta(num 1682405624501) 2023-04-25 06:56:43,414 DEBUG [RS:0;jenkins-hbase4:41021] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/oldWALs 2023-04-25 06:56:43,414 INFO [RS:0;jenkins-hbase4:41021] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41021%2C1682405622759:(num 1682405624346) 2023-04-25 06:56:43,414 DEBUG [RS:0;jenkins-hbase4:41021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:43,414 INFO [RS:0;jenkins-hbase4:41021] regionserver.LeaseManager(133): Closed leases 2023-04-25 06:56:43,415 INFO [RS:0;jenkins-hbase4:41021] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-25 06:56:43,415 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-25 06:56:43,416 INFO [RS:0;jenkins-hbase4:41021] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41021 2023-04-25 06:56:43,422 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41021,1682405622759 2023-04-25 06:56:43,422 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 06:56:43,422 ERROR [Listener at localhost/41663-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b0d86bd rejected from java.util.concurrent.ThreadPoolExecutor@f321304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2396] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 06:56:43,423 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 06:56:43,423 ERROR [Listener at localhost/41663-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c6f446c rejected from java.util.concurrent.ThreadPoolExecutor@f321304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2396] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 06:56:43,424 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41021,1682405622759] 2023-04-25 06:56:43,424 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41021,1682405622759; numProcessing=1 2023-04-25 06:56:43,426 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41021,1682405622759 already deleted, retry=false 2023-04-25 06:56:43,426 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41021,1682405622759 expired; onlineServers=0 2023-04-25 06:56:43,426 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36703,1682405621453' ***** 2023-04-25 06:56:43,426 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-25 06:56:43,426 DEBUG [M:0;jenkins-hbase4:36703] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5524377e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-25 06:56:43,426 INFO [M:0;jenkins-hbase4:36703] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36703,1682405621453 2023-04-25 06:56:43,426 INFO [M:0;jenkins-hbase4:36703] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36703,1682405621453; all regions closed. 2023-04-25 06:56:43,426 DEBUG [M:0;jenkins-hbase4:36703] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 06:56:43,426 DEBUG [M:0;jenkins-hbase4:36703] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-25 06:56:43,426 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-25 06:56:43,426 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682405623997] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682405623997,5,FailOnTimeoutGroup] 2023-04-25 06:56:43,426 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682405623996] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682405623996,5,FailOnTimeoutGroup] 2023-04-25 06:56:43,426 DEBUG [M:0;jenkins-hbase4:36703] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-25 06:56:43,427 INFO [M:0;jenkins-hbase4:36703] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-25 06:56:43,427 INFO [M:0;jenkins-hbase4:36703] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-25 06:56:43,427 INFO [M:0;jenkins-hbase4:36703] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-25 06:56:43,428 DEBUG [M:0;jenkins-hbase4:36703] master.HMaster(1512): Stopping service threads 2023-04-25 06:56:43,428 INFO [M:0;jenkins-hbase4:36703] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-25 06:56:43,428 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-25 06:56:43,428 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 06:56:43,428 ERROR [M:0;jenkins-hbase4:36703] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-25 06:56:43,429 INFO [M:0;jenkins-hbase4:36703] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-25 06:56:43,429 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-25 06:56:43,429 DEBUG [M:0;jenkins-hbase4:36703] zookeeper.ZKUtil(398): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-25 06:56:43,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 06:56:43,429 WARN [M:0;jenkins-hbase4:36703] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-25 06:56:43,429 INFO [M:0;jenkins-hbase4:36703] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-25 06:56:43,429 INFO [M:0;jenkins-hbase4:36703] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-25 06:56:43,430 DEBUG [M:0;jenkins-hbase4:36703] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-25 06:56:43,430 INFO [M:0;jenkins-hbase4:36703] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:56:43,430 DEBUG [M:0;jenkins-hbase4:36703] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:56:43,430 DEBUG [M:0;jenkins-hbase4:36703] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-25 06:56:43,430 DEBUG [M:0;jenkins-hbase4:36703] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:56:43,430 INFO [M:0;jenkins-hbase4:36703] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.84 KB heapSize=393.59 KB 2023-04-25 06:56:43,444 INFO [M:0;jenkins-hbase4:36703] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.84 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e1b45c8a9cc24f0494af9504c81d8620 2023-04-25 06:56:43,447 INFO [M:0;jenkins-hbase4:36703] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e1b45c8a9cc24f0494af9504c81d8620 2023-04-25 06:56:43,448 DEBUG [M:0;jenkins-hbase4:36703] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e1b45c8a9cc24f0494af9504c81d8620 as hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e1b45c8a9cc24f0494af9504c81d8620 2023-04-25 06:56:43,451 INFO [M:0;jenkins-hbase4:36703] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e1b45c8a9cc24f0494af9504c81d8620 2023-04-25 06:56:43,451 INFO [M:0;jenkins-hbase4:36703] regionserver.HStore(1080): Added hdfs://localhost:44945/user/jenkins/test-data/2e7d3f4d-c9bd-9cd7-492b-2746ae309767/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e1b45c8a9cc24f0494af9504c81d8620, entries=65, sequenceid=578, filesize=24.7 K 2023-04-25 06:56:43,452 INFO [M:0;jenkins-hbase4:36703] regionserver.HRegion(2948): Finished flush of dataSize ~344.84 KB/353121, heapSize ~393.58 KB/403024, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=578, compaction requested=false 2023-04-25 06:56:43,453 INFO [M:0;jenkins-hbase4:36703] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 06:56:43,453 DEBUG [M:0;jenkins-hbase4:36703] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 06:56:43,455 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-25 06:56:43,455 INFO [M:0;jenkins-hbase4:36703] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-25 06:56:43,456 INFO [M:0;jenkins-hbase4:36703] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36703 2023-04-25 06:56:43,458 DEBUG [M:0;jenkins-hbase4:36703] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,36703,1682405621453 already deleted, retry=false 2023-04-25 06:56:43,525 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 06:56:43,525 INFO [RS:0;jenkins-hbase4:41021] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41021,1682405622759; zookeeper connection closed. 2023-04-25 06:56:43,525 ERROR [Listener at localhost/41663-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64059773 rejected from java.util.concurrent.ThreadPoolExecutor@f321304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2396] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 06:56:43,526 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): regionserver:41021-0x101ba1865200001, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 06:56:43,526 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@de70924] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@de70924 2023-04-25 06:56:43,526 ERROR [Listener at localhost/41663-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4cb51ec2 rejected from java.util.concurrent.ThreadPoolExecutor@f321304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2396] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 06:56:43,527 INFO [Listener at localhost/41663] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-25 06:56:43,625 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 06:56:43,625 INFO [M:0;jenkins-hbase4:36703] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36703,1682405621453; zookeeper connection closed. 2023-04-25 06:56:43,625 ERROR [Listener at localhost/41663-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@44d39f93 rejected from java.util.concurrent.ThreadPoolExecutor@3f5c750[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4002] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 06:56:43,625 DEBUG [Listener at localhost/41663-EventThread] zookeeper.ZKWatcher(600): master:36703-0x101ba1865200000, quorum=127.0.0.1:60416, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 06:56:43,625 ERROR [Listener at localhost/41663-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6de45497 rejected from java.util.concurrent.ThreadPoolExecutor@3f5c750[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4002] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 06:56:43,627 WARN [Listener at localhost/41663] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-25 06:56:43,630 INFO [Listener at localhost/41663] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-25 06:56:43,739 WARN [BP-1734206070-172.31.14.131-1682405618087 heartbeating to localhost/127.0.0.1:44945] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-25 06:56:43,739 WARN [BP-1734206070-172.31.14.131-1682405618087 heartbeating to localhost/127.0.0.1:44945] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1734206070-172.31.14.131-1682405618087 (Datanode Uuid 4823a6af-3343-4f18-aae6-ee75d58162a4) service to localhost/127.0.0.1:44945 2023-04-25 06:56:43,742 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/cluster_91fce01a-9db1-5e0b-1ff6-3a8bbb37f430/dfs/data/data1/current/BP-1734206070-172.31.14.131-1682405618087] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-25 06:56:43,742 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9d1aee05-3106-d450-4f27-b29e3e35b753/cluster_91fce01a-9db1-5e0b-1ff6-3a8bbb37f430/dfs/data/data2/current/BP-1734206070-172.31.14.131-1682405618087] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-25 06:56:43,779 INFO [Listener at localhost/41663] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-25 06:56:43,944 INFO [Listener at localhost/41663] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-25 06:56:44,138 INFO [Listener at localhost/41663] hbase.HBaseTestingUtility(1293): Minicluster is down